More Progress!

I resurrected the test, edoc, clean, and release tasks tonight. Everything works as it should with the exception of edoc. There seems to be a conflict between edoc and xmerl in the newest version of erlang (R11B-3). I hope to figure out what the problem is and resolve it soon.

On a side note. I came across the cover module in the tools application of Erlang. Since I was working on the unit tests task at the time it seemed like a good idea to add this functionality to the build system either via a code coverage task or integration into the existing unit tests task. The only real issue is that all the dependent modules need to be recompiled with special coverage information to make cover work. This adds quite a bit of complexity to the feature. In any case, its on my list of new tasks to add.

I added one task additional, an analyze tasks. It seems the dialyzer folks finally made it controllable from erlang code. This allowed me to integrate dialyzer into sinan. Now running dialyzer will be as simple as running a build task. Hopefully, if dialyzer is easier to use it will get used more often.

Building Again!

I finally finished refactoring the build task tonight. For the first time the latest version of the build system is actually building source. It took a huge amount of effort to get the dependent tasks to this point. The engine, discover, depends, the repo puller, all of it was just preparation for the builder. In any case, I am very pleased.

For the moment it just builds *.erl and *.yrl but eventually I want to support all of the erlang compilables. I suspect that refactoring the rest of the tasks (test, release, tar, clean, etc) wont take much longer. Hopefully, I will be able to do an alpha release at the end of this week. I certainly hope so in any case.

Topological Sort Joy

I finally modified sinan to make use of all the dependency analysis code I wrote for ewrepo. One thing that I needed to do was produce a list, in the dependent order, of the graph of dependencies in the internal project applications. This is essential for compile time things like parse transforms are going to work correctly. I wrestled with how to do this correctly for quite a bit before I realized that its a simple topological sort. I should really have realized this immediately, but I didn't. Fortunately, this isn't a difficult algorithm, especially since I didn't even need to implement it. Joe Armstrong wrote a topo sort for his ermake project and made it available in the contribs section of erlang.org. Once I found that it didn't take to long to integrate it into sinan.

Once I got to thinking about this dependency problem I realized that there was at least one more area that a topological sort was needed. Thats in the run order of the task list. Each task may have any number of tasks that it depends on. Right now my code would just run each dependent task multiple times. Thats a bug, fortunately it wont take more then a few minutes to integrate the topo sort into this area as well.

Dependency Checking Complete!!

I finally finished up both the shallow and the deep dependency resolution for sinan. This took a bit of time and effort to conceptualize, abstract and implement.

Now given a list of apps and the dependencies (along with the url(s) of the repo); like this

  [{app1, "0.1", [{edoc, 'LATEST'},                                                                        
                 {syntax_tools, "1.5.0"}]}

It correctly returns a list of all the dependencies required for the project; like this

 [{stdlib, "1.14.2",
           [compiler,edoc,syntax_tools]},
  {syntax_tools,"1.5.0",
           [app1,edoc]},
  {compiler,"4.4.2",
           [edoc]},
  {kernel,"2.11.2",
           [compiler,stdlib,edoc]},
  {edoc,"0.6.9",
           [app1]},
  {app1,"0.1",[]}]    

If it runs into a version conflict it reports what the conflict is and what applications are causing it.

In the output above. Each entry is a dependency, its version and a list of apps the depend on that app.

Released: Ktuo 0.1.1

I just released the first alpha version of my JSON Parser/Encoder Ktuo. I use it in several projects with no problems at all. However, this is still an alpha release, so be aware that you may run into issues. The download area for Ktuo is here. A bit of documentation for the project is here.

I know the question of why another JSON parser is going to come up, so I am just going to go ahead and address it now. The existing JSON parser is still available and still useful. It makes elegant use of continuations to handle cases where it may not have a full JSON expression available. This makes it very flexible and pretty darn interesting. However, these extra features also add significant complexity and no small amount of additional resource usage. While working on my Tercio project I realized that I didn't need these features and I didn't want to pay the cost of the complexity and resource usage overhead. So I wrote my own. Eventually, I realized it was useful in and of itself and created a project space for it.

Finished Up Dependency Checking

I finished up the shallow dependency checking. Now I just need to integrate with the repository and repository metadata to do deep dependency analysis and package pulling.

More Dependencies

Most people think that Erlang has just runtime dependencies. For
the most part this is true. However, Erlang also has two types of
compile time dependencies. The first is the ".hrl" files that are
included into a module. The second type is ".erl" files that
implement a parse transform. Both of these types of files
represent compile time dependencies. For the most part its simple
enough to make sure the include and code path information is
set. However, its much more difficult to make sure that if one of
these types of files change all of its dependencies are changed
as well. I think that the easiest way to handle this would be to
simply look at the ast of a compiled file during the compilation
process and extract the includes and the parse transform
information from that ast. Unfortunately, it means knowing much
more about a dependent OTP application then the build system
currently knows. It would definitely need to happen after the high
level runtime dependency mapping takes place. I think, for now, I
will stick with just handling the runtime dependencies and making
sure that the build time dependencies have the right path
information available. Once the system gets out in the wild and I
start getting feedback I may modify it to be a little smarter
about the compile time dependencies.

Dependency Detection

I am working on dependency detection for erlware (sinan and the erlware build system) today. Its going fairly well but I think I can significantly reduce the complexity. Any any case, I will put this new code in ewrepo as soon as we get that code out somewhere.

Per App Build Config

I have added the ability to configure the build directives for an application on a per app basis. This should make it much easier to do custom builds. Of course, for those of apps that don't need per app info it can be left off. This is a big deal and one of the bigger requests I have gotten.

Config Refactoring Complete!

I just finished refactoring the config support in sinan using the new ktuo package. It works really well and makes the config much more readable.

Original config with erlang terms

[{repositories, ["http://repo.metadrift.net/repo"]},
{build_dir, "_build"},
{ignore_dirs, ["_", "."]},
{ignore_apps, []},
{default_task,  build},
{tasks, [{discover,
          [{handler, sin_discover}]},
       {verify, [{depends, [discover]},
                 {handler, sin_verify}]},
       {build, [{depends, [verify]},
                {handler, sin_erl_builder}]},
       {doc, [{depends, [build]},
              {handler, sin_edoc}]},
       {tar, [{depends, [build]},
              {handler, sin_tar}]}]}].

new, more readable, config

repositories : ["http://repo.metadrift.net/repo"],

build_dir : _build,

ignore_dirs : ["_",
           "."],

ignore_apps : [],

default_task :  build,

tasks : {

discover : {
  handler : sin_discover
},

verify : {
  depends : [discover],
  handler : sin_verify
},

build : {
  depends : [verify],
  handler : sin_erl_builder
},

doc : {
  depends : [build],
  handler : sin_edoc
},

tar : {
  depends : [build],
  handler : sin_tar
}

}

Well, I think its more readable and more maintainable with the nested namespaces.

Ktuo Error Reporting

I have vastly increased the usefulness of error reporting in ktuo. Before it just failed with badmatch if an error occurred. Now it gives a useful error message along with the line and column number of the problem. Now that I am using the library as a config parsing lib in a couple of places it made sense that it report error usefully.

JSON Parser

In the course of developing tercio I built a fast little json parser/encoder. Well I decided to use it in sinan as well so I moved it to its own project. The project, ktuo, is hosted in the usual place. It outputs strict json, but it parses a small superset of json. It allows single words as atoms in the json string as an alternative to a single word enclosed in quotes. This allows the lib to be used as a config parser. It makes the config much more readable. In any case, the library is usable and available now.

Sinan Build System

I have been working on a build system for erlang for some time. I have a working version that I have used. However, it has a couple of shortcomings that rendered it difficult to use. I done some significant work towards version two. I decided to make the source available via google code hosting anyway. There is a large amount of proprietary software and systems going on in the community right now and I wanted to be in the open from the start.

the source is here.

Its named after Ḳoca Mi‘mār Sinān Āġā one of the great architects and builders of the Ottoman Empire. I thought it appropriate to name a build system after him.

JsPkg Discarded, Package Support in Place

I decided to not use jspkg. The code there needs a massive refactoring and it approaches the problem from a viewpoint that is very different from my own. In stead, I wrote a simple packager that doesn't do loading. So for now I have good namespacing support and stubs for package loading. Once I get a better understanding of the semantics of the applications built on this platform I will start filling out the loading stubs.

Into the Client

Well I have moved into client side code. To save a little time I am building the client side code on top of jspkg. It needs quite a bit of refactoring as well as some conversion to work on top of prototype. However, it should fill my needs nicely. Once the package stuff is in place I can build up a nice messaging layer.