While trying to generalize our practices from KendoUI related projects we've participated so far, we updated control.js - a small javascript additions to KendoUI.

At present we have defined:

1. An extended model. See KendoUI extended model.

2. A lightweight user control - a widget to bind a template and a model, and to facilitate declarative instantiation. See KendoUI User control.

3. A reworked version of nesterovskyBros.defineControl() function.

var widgetType = scope.defineControl(
{
   name: widget-name-string,
   model: widget-model-type,
   template: optional-content-template,
   windowOptions: optional-window-options
},
base);

When optional-content-template is not specified then template is calculated as following:

var template = options.temlate || proto.template || model.temlate;

if (template === undefined)
{
  template = scope.template(options.name.toLowerCase() + "-template");
}

When windowOptions is specified then widgetType.dialog(options) function is defined. It's used to open dialog based on the specified user control. windowOptions is passed to kendo.ui.Window constructor. windowOptions.closeOnEscape indicates whether to close opened dialog on escape.

widgetType.dialog() returns a kendo.ui.Window instance with content based on the user control. Window instance contains functions:

• result() - a $.Deffered for the dialog result, and
• model() - referring to the user control model.

 The model instance has functions:

•  dialog() referring to the dialog, and
• result() referring to the dialog result.

widget.dialog() allows all css units in windowOptions.width and windowOptions.height parameters.

base - is optional user control base. It defaults to nesterovskyBros.ui.UserControl.

4. Adjusted splitter. See Adjust KendoUI Splitter.

5. Auto resize support.

Layout is often depends on available area. One example is Splitter widget that recalculates its panes when window or container Splitter is resized. There are other cases when you would like to adjust layout when a container's area is changed like: adjust grid, tab, editor or user's control contents.

KendoUI does not provide a solution for this problem, so we have defined our own.

• A widget can be marked with class="auto-resize" marker;
• A widget may define a widgetType.autoResize(element) function that adapts widget to a new size.
• A code can call nesterovskyBros.resize(element) function at trigger resizing of the subtree.

To support existing controls we have defined autoResize() function for Grid, Splitter, TabStrip, and Editor widgets.

To see how auto resizing works, it's best to look into index.html, products.tmpl.html, and into the implementation controls.js.

Please note that we consider controls.js as an addition to KendoUI library. If in the future the library will integrate or implement similar features we will be happy to start using their API.

This time we update csharpxom to adjust it to C# 4.5. Additions areasync modifier and await operator.

They are used to simplify asynchronous programming.

The following example from the msdn:

private async Task<byte[]> GetURLContentsAsync(string url)
{
var content = new MemoryStream();
var request = (HttpWebRequest)WebRequest.Create(url);

using(var response = await request.GetResponseAsync())
using(var responseStream = response.GetResponseStream())
{
await responseStream.CopyToAsync(content);
}

return content.ToArray();
}

looks like this in csharpxom:

<method name="GetURLContentsAsync" access="private" async="true">
<returns>
<type name="Task" namespace="System.Threading.Tasks">
<type-arguments>
<type name="byte" rank="1"/>
</type-arguments>
</type>
</returns>
<parameters>
<parameter name="url">
<type name="string"/>
</parameter>
</parameters>
<block>
<var name="content">
<initialize>
<new-object>
<type name="MemoryStream" namespace="System.IO"/>
</new-object>
</initialize>
</var>
<var name="request">
<initialize>
<cast>
<invoke>
<static-method-ref name="Create">
<type name="WebRequest" namespace="System.Net"/>
</static-method-ref>
<arguments>
<var-ref name="url"/>
</arguments>
</invoke>
<type name="HttpWebRequest" namespace="System.Net"/>
</cast>
</initialize>
</var>

<using>
<resource>
<var name="response">
<initialize>
<await>
<invoke>
<method-ref name="GetResponseAsync">
<var-ref name="request"/>
</method-ref>
</invoke>
</await>
</initialize>
</var>
</resource>
<using>
<resource>
<var name="responseStream">
<initialize>
<invoke>
<method-ref name="GetResponseStream">
<var-ref name="response"/>
</method-ref>
</invoke>
</initialize>
</var>
</resource>
<expression>
<await>
<invoke>
<method-ref name="CopyToAsync">
<var-ref name="responseStream"/>
</method-ref>
<arguments>
<var-ref name="content"/>
</arguments>
</invoke>
</await>
</expression>
</using>
</using>

<return>
<invoke>
<method-ref name="ToArray">
<var-ref name="content"/>
</method-ref>
</invoke>
</return>
</block>
</method>

Some time ago our younger brother Aleksander had started studying of cinematography.

Few days ago he started his own "multimedia" blog (you'll better understand me when you'll see it), where you can see his portfolio. Aleksander's latest work was made with cooperation with Ilan Lahov, see "Bar mitzvah". This work demonstrates Aleksander's progress in this field.

Our congratulations to Aleksander!

Please welcome a new human being Masha Vladimirovna Nesterovsky!

Recently one of users of java yield return annotation has kindly informed us about some problem that happened in his environment (see Java's @Yield return annotation update).

Incidentally we have never noticed the problem earlier. Along with this issue we have found that eclipse compiler has changed in the Indigo in a way that we had to recompile the source. Well, that's a price you have to pay when you access internal API.

Updated sources can be found at Yield.zip, and compiled jars at Yield.jar (pre-Indigo), and Yield.3.7.jar (Indigo and probably higher).

See also:

Yield return feature in java
Why @Yield iterator should be Closeable
What you can do with jxom.

We did not update languages-xom already for many monthes but now we have found a severe bug in the jxom's algorithm for eliminating unreachable code. The marked line were considered as unreachable:

check:
if (condition)
{
break check;
}
else
{
return;
}

// due to bug the following was considered unreachable
expression;

Bug is fixed.

Current update contains other cosmetic fixes.

Please download xslt sources from languages-xom.zip.

Summary

Languages XOM is a set of xml schemas and xslt stylesheets that allows:

• to define programs in xml form;
• to perform transformations over code in xml form;
• to generate sources.

Languages XOM includes:

• jxom - Java Xml Object model;
• csharpxom - C# Xml Object Model;
• cobolxom - COBOL Xml Object Model;
• sqlxom - SQL Xml Object Model (including several sql dialects);
• aspx - ASP.NET Object Model;

A proprietary part of languages XOM also includes XML Object Model for a language named Cool:GEN. In fact the original purpose for this API was a generation of java/C#/COBOL from Cool:GEN. For more details about Cool:GEN conversion please see here.

Ladies and gentlemen!

We are proud and would like to announce few works of our younger brother Aleksander, who studies cinematography now.

We have updated @Yield annotation processor to support better debug info.

Annotation processor can be downloaded from Yield.zip or Yield.jar.

We also decided to consider jxom's state machine refactoring as obsolete as @Yield annotation allows to achieve the same effect but with more clear code.

JXOM can be downloaded from languages-xom.zip

See also:

Yield return feature in java
Why @Yield iterator should be Closeable
What you can do with jxom.

We're happy to announce that we have implemented @Yield annotation both in javac and in eclipse compilers.

This way you get built-in IDE support for the feature!

To download yield annotation processor please use the following link: Yield.zip

It contains both yield annotation processor, and a test project.

If you do not want to compile the sources, you can download Yield.jar

We would like to reiterate on how @Yield annotation works:

1. A developer defines a method that returns either Iterator<T> or Iterable<T> instance and marks it with @Yield annotation.
2. A developer implements iteration logic following the pattern:
   
   • declare a variable to accumulate results:
     ArrayList<T> items = new ArrayList<T>();
   • use the following statement to add item to result:
     items.add(...);
   • use
     return items;
     or
     return items.iterator();
to return result;
   • mark method's params, if any, as final.
3. A devoloper ensures that yield annotation processor is available during compilation (see details below).
4. YieldProcessor rewrites method into a state machine at compilation time.

The following is an example of such a method:

@Yield
public static Iterable<Integer> generate(final int from, final int to)
{
ArrayList<Integer> items = new ArrayList<Integer>();

for(int i = from; i < to; ++i)
{
items.add(i);
}

return items;
}

The use is like this:

for(int value: generate(7, 20))
{
System.out.println("generator: " + value);
}

Notice that method's implementation still will be correct in absence of YieldProcessor.

Other important feature is that the state machine returned after the yield processor is closeable.

This means that if you're breaking the iteration before the end is reached you can release resources acquired during the iteration.

Consider the example where break exits iteration:

@Yield
public static Iterable<String> resourceIteration()
{
ArrayList<String> items = new ArrayList<String>();

acquire();

try
{
for(int i = 0; i < 100; ++i)
{
items.add(String.valueOf(i));
}
}
finally
{
release();
}

return items;
}

and the use

int i = 0;
Iterable<String> iterator = resourceIteration();

try
{
for(String item: iterator)
{
System.out.println("item " + i + ":" + item);

if (i++ > 30)
{
break;
}
}
}
finally
{
close(iterator);
}

...

private static <T> void close(T value)
throws IOException
{
if (value instanceof Closeable)
{
Closeable closeable = (Closeable)value;

closeable.close();
}
}

Close will execute all required finally blocks. This way resources will be released.

To configure yield processor a developer needs to refer Yield.jar in build path, as it contains @Yield annotation. For javac it's enough, as compiler will find annotation processor automatically.

Eclipse users need to open project properties and:

• go to the "Java Compiler"/"Annotation Processing"
• mark "Enable project specific settings"
• select "Java Compiler"/"Annotation Processing"/"Factory Path"
• mark "Enable project specific settings"
• add Yield.jar to the list of "plug-ins and JARs that contain annotation processors".

At the end we want to point that @Yield annotation is a syntactic suggar, but it's important the way the foreach statement is important, as it helps to write concise and an error free code.

See also
Yield feature in java implemented!
Yield feature in java

For some reason we never knew about instance initializer in java; on the other hand static initializer is well known.

class A
{
int x;
static int y;

// This is an instance initializer.
{
x = 1;
}

// This is a static initializer.
static
{
y = 2;
}
}

Worse, we have missed it in the java grammar when we were building jxom. This way jxom was missing the feature.

Today we fix the miss and introduce a schema element:

<class-initializer static="boolean">
<block>
...
</block>
</class-initializer>

It superseeds:

<static>
<block>
...
</block>
</static>

that supported static initializers alone.

Please update languages-xom xslt stylesheets.

P.S. Out of curiosity, did you ever see any use of instance initializers?

We could not stand the temptation to implement the @Yield annotation that we described earlier.

Idea is rather clear but people are saying that it's not an easy task to update the sources.

They were right!

Implementation has its price, as we were forced to access JDK's classes of javac compiler. As result, at present, we don't support other compilers such as EclipseCompiler. We shall look later what can be done in this area.

At present, annotation processor works perfectly when you run javac either from the command line, from ant, or from other build tool.

Here is an example of how method is refactored:

@Yield
public static Iterable<Long> fibonachi()
{
ArrayList<Long> items = new ArrayList<Long>();

long Ti = 0;
long Ti1 = 1;

while(true)
{
items.add(Ti);

long value = Ti + Ti1;

Ti = Ti1;
Ti1 = value;
}
}

And that's how we transform it:

@Yield()
public static Iterable<Long> fibonachi() {
assert (java.util.ArrayList<Long>)(ArrayList<Long>)null == null : null;

class $state$ implements java.lang.Iterable<Long>, java.util.Iterator<Long>, java.io.Closeable {

public java.util.Iterator<Long> iterator() {
if ($state$id == 0) {
$state$id = 1;
return this;
} else return new $state$();
}

public boolean hasNext() {
if (!$state$nextDefined) {
$state$hasNext = $state$next();
$state$nextDefined = true;
}

return $state$hasNext;
}

public Long next() {
if (!hasNext()) throw new java.util.NoSuchElementException();

$state$nextDefined = false;

return $state$next;
}

public void remove() {
throw new java.lang.UnsupportedOperationException();
}

public void close() {
$state$id = 5;
}

private boolean $state$next() {
while (true) switch ($state$id) {
case 0:
$state$id = 1;
case 1:
Ti = 0;
Ti1 = 1;
case 2:
if (!true) {
$state$id = 4;
break;
}

$state$next = Ti;
$state$id = 3;

return true;
case 3:
value = Ti + Ti1;
Ti = Ti1;
Ti1 = value;
$state$id = 2;

break;
case 4:
case 5:
default:
$state$id = 5;

return false;
}
}

private long Ti;
private long Ti1;
private long value;
private int $state$id;
private boolean $state$hasNext;
private boolean $state$nextDefined;
private Long $state$next;
}

return new $state$();
}

Formatting is automatic, sorry, but anyway it's for diagnostics only. You will never see this code.

It's iteresting to say that this implementation is very precisely mimics xslt state machine implementation we have done back in 2008.

You can download YieldProcessor here. We hope that someone will find our solution very interesting.

You might be interested in the following article that was written in form of a little guide. It can educate about new ways to learn SQL and hopefully may help someone to get a job. See "How to get MS SQL certification" that was written by Michele Rouse.

Historically jxom was developed first, and as such exhibited some imperfectness in its xml schema. csharpxom has taken into an account jxom's problems.

Unfortunately we could not easily fix jxom as a great amount of code already uses it. In this refactoring we tried to be conservative, and have changed only "type" and "import" xml schema elements in java.xsd.

Consider type reference and package import constructs in the old schema:

<!-- import java.util.ArrayList; -->
<import name="java.util.ArrayList"/>

<!-- java.util.ArrayList<java.math.BigDecimal> -->
<type package="java.util">
<part name="ArrayList">
<argument>
<type name="BigDecimal" package="java.math">
</argument>
</part>
</type>

<!-- my.Parent.Nested -->
<type package="my">
<part name="Parent"/>
<part name="Nested"/>
<type>

Here we can observe that:

• type is referred by a qualified name in import element;
• type has two forms: simple (see BigDecimal), and other for nested or generic type (see ArrayList).

We have made it more consistent in the updated jxom:

<!-- import java.util.ArrayList; -->
<import>
<type name="ArrayList" package="java.util"/>
</import>

<!-- java.util.ArrayList<java.math.BigDecimal> -->
<type name="ArrayList" package="java.util">
<argument>
<type name="BigDecimal" package="java.math">
</argument>
</type>

<!-- my.Parent.Nested -->
<type name="Nested">
<type name="Parent" package="my"/>
<type>

We hope that you will not be impacted very much by this fix.

Please refresh Languages XOM from languages-xom.zip.

P.S. we have also included xml schema and xslt api to generate ASPX (see Xslt serializer for ASPX output). We, in fact, in our projects, generate aspx documents with embedded csharpxom, and then pass it through two stage transformation.

In the previous post we have announced an API to parse a COBOL source into the cobolxom.

We exploited the incremental parser to build a grammar xml tree and then were planning to create an xslt transformation to generate cobolxom.

Now, we would like to declare that such xslt is ready.

At present all standard COBOL constructs are supported, but more tests are required. Preprocessor support is still in the todo list.

You may peek into an examples of COBOL:

Cobol grammar:

And cobolxom:

While we were building a grammar to cobolxom stylesheet we asked ourselves whether the COBOL parsing could be done entirely in xslt. The answer is yes, so who knows it might be that we shall turn this task into pure xslt one. :-)

Once ago we have created an incremental parser, and now when we have decided to load COBOL sources directly into cobolxom (XML Object Model for a COBOL) the parser did the job perfectly.

The good point about incremental parser is that it easily handles COBOL's grammar.

The whole process looks like this:

1. incremental parser having a COBOL grammar builds a grammar tree;
2. we stream this tree into xml;
3. xslt to transform xml from previous step into cobolxom (TODO).

This is an example of a COBOL:

IDENTIFICATION DIVISION.
PROGRAM-ID. FACTORIAL RECURSIVE.

DATA DIVISION.
WORKING-STORAGE SECTION.
01 NUMB PIC 9(4) VALUE IS 5.
01 FACT PIC 9(8) VALUE IS 0.

LOCAL-STORAGE SECTION.
01 NUM PIC 9(4).

PROCEDURE DIVISION.
MOVE 'X' TO XXX
MOVE NUMB TO NUM

IF NUMB = 0 THEN
MOVE 1 TO FACT
ELSE
SUBTRACT 1 FROM NUMB
CALL 'FACTORIAL'
MULTIPLY NUM BY FACT
END-IF

DISPLAY NUM '! = ' FACT

GOBACK.
END PROGRAM FACTORIAL.

And a grammar tree:

<Program>
<Name data="FACTORIAL"/>
<Recursive/>
<DataDivision>
<WorkingStorageSection>
<Data>
<Level data="01"/>
<Name data="NUMB"/>
<Picture data="9(4)"/>
<Value>
<Numeric data="5"/>
</Value>
</Data>
<Data>
<Level data="01"/>
<Name data="FACT"/>
<Picture data="9(8)"/>
<Value>
<Numeric data="0"/>
</Value>
</Data>
</WorkingStorageSection>
<LocalStorageSection>
<Data>
<Level data="01"/>
<Name data="NUM"/>
<Picture data="9(4)"/>
</Data>
</LocalStorageSection>
</DataDivision>
<ProcedureDivision>
<Sentence>
<MoveStatement>
<From>
<String data="'X'"/>
</From>
<To>
<Identifier>
<DataName data="XXX"/>
</Identifier>
</To>
</MoveStatement>
<MoveStatement>
<From>
<Identifier>
<DataName data="NUMB"/>
</Identifier>
</From>
<To>
<Identifier>
<DataName data="NUM"/>
</Identifier>
</To>
</MoveStatement>
<IfStatement>
<Condition>
<Relation>
<Identifier>
<DataName data="NUMB"/>
</Identifier>
<Equal/>
<Numeric data="0"/>
</Relation>
</Condition>
<Then>
<MoveStatement>
<From>
<Numeric data="1"/>
</From>
<To>
<Identifier>
<DataName data="FACT"/>
</Identifier>
</To>
</MoveStatement>
</Then>
<Else>
<SubtractStatement>
<Value>
<Numeric data="1"/>
</Value>
<From>
<Identifier>
<DataName data="NUMB"/>
</Identifier>
</From>
</SubtractStatement>
<CallStatement>
<Name>
<String data="'FACTORIAL'"/>
</Name>
</CallStatement>
<MultiplyStatement>
<Value>
<Identifier>
<DataName data="NUM"/>
</Identifier>
</Value>
<By>
<Identifier>
<DataName data="FACT"/>
</Identifier>
</By>
</MultiplyStatement>
</Else>
</IfStatement>
<DisplayStatement>
<Values>
<Identifier>
<DataName data="NUM"/>
</Identifier>
<String data="'! = '"/>
<Identifier>
<DataName data="FACT"/>
</Identifier>
</Values>
</DisplayStatement>
<GobackStatement/>
</Sentence>
</ProcedureDivision>
<EndName data="FACTORIAL"/>
</Program>

The last step is to transform tree into cobolxom is in the TODO list.

We have commited COBOL grammar in the same place at SourceForge as it was with XQuery grammar. Solution is now under the VS 2010.

Recently we were raising a question about serialization of ASPX output in xslt.

The question went like this:

What's the recommended way of ASPX page generation?
E.g.:

------------------------
 <%@ Page AutoEventWireup="true"
   CodeBehind="CurMainMenuP.aspx.cs"
   EnableSessionState="True"
   Inherits="Currency.CurMainMenuP"
   Language="C#"
   MaintainScrollPositionOnPostback="True"
   MasterPageFile="Screen.Master" %>

<asp:Content ID="Content1" runat="server" ContentPlaceHolderID="Title">CUR_MAIN_MENU_P</asp:Content>

<asp:Content ID="Content2" runat="server" ContentPlaceHolderID="Content">
  <span id="id1222146581" runat="server"
    class="inputField system UpperCase" enableviewstate="false">
    <%# Dialog.Global.TranCode %>
  </span>
  ...
------------------------

Notice aspx page directives, data binding expessions, and prefixed tag names without namespace declarations.

There was a whole range of expected answers. We, however, looked whether somebody have already dealed with the task and has a ready solution at hands.

In general it seems that xslt community is very angry about ASPX: both format and technology. Well, put this aside.

The task of producing ASPX, which is almost xml, is not solvable when you're staying with pure xml serializer. Xslt's xsl:character-map does not work at all. In fact it looks as a childish attempt to address the problem, as it does not support character escapes but only grabs characters and substitutes them with strings.

We have decided to create ASPX serializer API producing required output text. This way you use <xsl:output method="text"/> to generate ASPX pages.

With this goal in mind we have defined a little xml schema to describe ASPX irregularities in xml form. These are:

• <xs:element name="declared-prefix"> - to describe known prefixes, which should not be declared;
• <xs:element name="directive"> - to describe directives like <%@ Page %>;
• <xs:element name="content"> - a transparent content wrapper;
• <xs:element name="entity"> - to issue xml entity;
• <xs:element name="expression"> - to describe aspx expression like <%# Eval("A") %>;
• <xs:element name="attribute"> - to describe an attribute of the parent element.

This approach greately simplified for us an ASPX generation process.

The API includes:

• aspx.xsd - an xml schema for the ASPX elements;
• aspx-serializer.xslt - a serializer API;
• aspx-test.xslt - a test stylesheet (any xml input is used).

We have implemented report parser in C#. Bacause things are spinned around C#, a schema definition is changed.

We have started from classes defining a report definition tree, annotated these classes for xml serialization, and, finally, produced xml schema for such tree. So, at present, it is not an xml schema with annotations but a separate xml schema.

In addition we have defined APIs:

• to enumerate report data (having report definition and report data one can get IEnumerable<ViewValue> to iterate report data in structured form);
• to read report through XmlReader, which allows, for example, to have report as input for an xslt tranformation.
• to write report directly into XmlWriter.

An example of report definition as C# code is: MyReport.cs. The very same report definition but serialized into xml is my-report.xml. A generated xml schema for a report definition is: schema0.xsd.

The good point about this solution is that it's already flexible enough to describe every report layout we have at hands, and it's extendable. Our measurments show that report parsing is extremely fast and have very small memory footprint due to forward only nature of report definitions.

From the design point of view report definition is a view of original text data with view info attached.

At present we have defined following views:

• Element - a named view to generate output from a content view;
•  Content - a view to aggregate other views together;
• Choice - a view to produce output from one of content views;
• Sequence - a view to sequence input view by key expressions, and to attach an index to each sequence item;
• Iterator - a view to generate output from input view while some condition is true, and to attach an iteration index to each part of output view;
• Page - a view to remove page headers and footers in the input view, and to attach an index to each page;
• Compute - a named view to produce result of evaluation of expression as output view;
•  Data - a named view to produce output value from some bounds of input view, and optionally to convert, validate and format the value.

To specify details of definitions there are:

• expressions to deal with integers: Add, Div, Integer, MatchProperty, Max, Min, Mod, Mul, Neg, Null, Sub, VariableRef, ViewProperty, Case;
• conditions to deal with booleans: And, EQ, GE, GT, IsMatch, LE, LT, NE, Not, Or.

At present there is no a specification of a report definitions. Probably, it's the most complex part to create such a spec for a user without deep knowledge. At present, our idea is that one should use xml schema (we should polish generated schema) for the report definition and schema aware editor to build report definitions. That's very robust approach working perfectly with languages xom.

C# sources can be found at: ReportLayout.zip including report definition classes and a sample report.

We're facing a task of parsing reports produced from legacy applications and converting them into a structured form, e.g. into xml. These xml files can be processed further with up to date tools to produce good looking reports.

Reports at hands are of very different structure and of size: from a couple of KB to a several GB. The good part is that they mostly have a tabular form, so it's easy to think of specific parsers in case of each report type.

Our goal is to create an environment where a less qualified person(s) could create and manage such parsers, and only rarely to engage someone who will handle less untrivial cases.

Our analysis has shown that it's possible to write such parser in almost any language: xslt, C#, java.

Our approach was to create an xml schema annotations that from one side define a data structure, and from the other map report layout. Then we're able to create an xslt that will generate either xslt, C#, or java parser according to the schema definitions. Because of languages xom, providing XML Object Model and serialization stylesheets for C# and java, it does not really matter what we shall generate xslt or C#/java, as code will look the same.

The approach we're going to use to describe reports is not as powerfull as conventional parsers. Its virtue, however, is simplicity of specification.

Consider a report sample (a data to extract is in bold):

1 TITLE ...                    PAGE:            1
 BUSINESS DATE: 09/30/09 ...
RUN DATE: 02/23/10 CYCLE : ITD RUN: 001 ... RUN TIME: 09:22:39 CM BUS ... CO NBR FRM
FUNC ... ----- ----- ----- ----- XXX065065CLR ... YYY ...
... 1 TITLE ... PAGE: 2 BUSINESS DATE: 09/30/09 ... RUN DATE: 02/23/10 CYCLE : ITD
RUN: 001 ... RUN TIME: 09:22:39 CM BUS ... CO NBR FRM FUNC ... ----- ----- ----- ----- AAANNNMMMPPP ... BBB ...
... * * * * * E N D O F R E P O R T * * * * *

We're approaching to the report through a sequence of views (filters) of this report. Each veiw localizes some report data either for the subsequent filterring or for the extraction of final data.

Looking into the example one can build following views of the report:

1. View of data before the "E N D   O F   R E P O R T" line.
2. View of remaining data without page headers and footers.
3. Views of table rows.
4. Views of cells.

A sequence of filters allows us to build a pipeline of transformations of original text. This also allows us to generate a clean xslt, C# or java code to parse the data.

At first, our favorite language for such parser was xslt. Unfortunatelly, we're dealing with Saxon xslt implementation, which is not very strong in streaming processing. Without a couple of extension functions to prevent caching, it tends to cache whole input in the memory, which is not acceptable.

At present we have decided to start from C# code, which is pure C# naturally. :-)

Code still is in the development but at present we would like to share the xml schema annotations describing report layout: report-mapping.xsd, and a sample of report description: test.xsd.

A few little changes in streaming and in name normalization algorithms in jxom and in csharpxom and the generation speed almost doubled (especially for big files).

We suspect, however, that our xslt code is tuned for saxon engine.

It would be nice to know if anybody used languages XOM with other engines. Is anyone using it at all (well, at least there are downloads)?

Languages XOM (jxom, csharpxom, cobolxom, sqlxom) can be loaded from: languages-xom.zip

When time has come to process big xml log files we've decided to implement streamable tree in saxon the very same way it was implemented in .net eight years ago (see How would we approach to streaming facility in xslt).

It's interesting enough that the implementation is similar to one of composable tree. There a node never stores a reference to a parent, while in the streamed tree no references to children are stored. This way only a limited subview of tree is available at any time. Implementation does not support preceding and preceding-sibling axes. Also, one cannot navigate to a node that is out of scope.

Implementation is external (there are no changes to saxon itself). To use it one needs to create an instance of DocumentInfo, which pulls data from XMLStreamReader, and to pass it as an input to a transformation:

Controller controller = (Controller)transformer;
XMLInputFactory factory = XMLInputFactory.newInstance();
StreamSource inputSource = new StreamSource(new File(input));
XMLStreamReader reader = factory.createXMLStreamReader(inputSource);
StaxBridge bridge = new StaxBridge();

bridge.setPipelineConfiguration(
  controller.makePipelineConfiguration());
bridge.setXMLStreamReader(reader);
inputSource = new DocumentImpl(bridge);

transformer.transform(inputSource, new StreamResult(output));

This helped us to format an xml log file of arbitrary size. An xslt like this can do the work:

<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns="http://www.w3.org/1999/xhtml"
  exclude-result-prefixes="xs">

  <xsl:template match="/log">
    <html>
      <head>
        <title>Log</title>
      </head>
      <body>
        <xsl:apply-templates/>
      </body>
    </html>
  </xsl:template>

  <xsl:template match="message">
   ...
  </xsl:template>

  <xsl:template match="message[@error]">
    ...
  </xsl:template>

  ...

</xsl:stylesheet>

Implementation can be found at: streamedtree.zip

Earlier, in the entry "Inline functions in xslt 2.1" we've described an implementation of xml tree model that may share subtrees among different trees.

This way, in a code:

<xsl:variable name="elements" as="element()*" select="..."/>

<xsl:variable name="result" as="element()">
  <result>
    <xsl:sequence select="$elements"/>
  </result>
</xsl:variable>

the implementation shares internal representation among $elements and subtree of $result. From the perspective of xslt it looks as completely different subtrees with different node identities, which is in the accordance with its view of the world.

After a short study we've decided to create a research implementation of this tree model in saxon. It's took only a couple of days to introduce a minimal changes to engine, to refactor linked tree into a new composable tree, and to perform some tests.

In many cases saxon has benefited immediately from this new tree model, in some other cases more tunings are required.

Our tests've showed that this new tree performed better than linked tree, but a little bit worser than tiny tree. On the other hand, it's obvious that conventional code patterns avoid subtree copying, assuming it's expensive operation, thus one should rethink some code practices to benefit from composable tree.

Implementation can be downloaded at: saxon.composabletree.zip

We've added a new language to the set of Xml Object Model schemas and stylesheets.

The newcomer is COBOL! No jokes. It's not a whim, really. Believe it or not but COBOL is still alive and we need to generate it (mainly different sorts of proxies).

We've used VS COBOL II grammar Version 1.0.3 as a reference. Implemented grammar is complete but without preprocessor statements. On the other hand it defines COPY and EXEC SQL constructs.

Definitely, it'll take a time for the xml schema and xslt implementation to become mature.

Now language XOM is:

• jxom - for java;
• csharpxom - for C#;
• cobolxom - for COBOL.

Sources can be found at languages-xom.

At present C# serializer knows how to print comments and do some formatting (we had to create micro xml serializer within xslt to serialize xml comments). C#'s formatting is not as advanced as java's one, but it should not be such in the first place, as C# text tends to be more neat due to properties and events. Compare:

Java: instance.getItems().get(10).setValue(value);

vs

C#: instance.Items[10].Value = value;

TODO: implement API existing in jxom and missing in C# xom. This includes:

• name normalization - rewriting tree to make names unique (duplicate names are often appear during generation from code templates);
• namespaces normalization - rewriting tree to elevate type namespaces (during generation, types are usually fully qualified);
• unreachable code detection - optional feature (in java it's required, as unreachable code is an error, while in C# it's only a warning);
• compile time expression evaluation - optional feature used in code optimization and in reachability checks;
• state machine refactoring - not sure, as C# has yield statement that does the similar thing.

Update can be found at: jxom/C# xom.

June, 24 update: name and namespace normalizations are implemented.

Writing a language serializer is an as easy task, as riding a bicycle. Once you learned it, you won't apply a mental force anymore to create a new one.

This still requires essential mechanical efforts to write and test things.

Well, this is the first draft of the C# xslt serializer. Archive contains both C# xom and jxom.

Note: no comments are still supported; nothing is done to format code except line wrapping.

Well, it's jxom no more but also csharpxom!

A project concerns demanded us to create a C# 3.0 xml schema.

Shortly we expect to create an xslt serializing an xml document in this schema into a text. Thankfully to the original design we can reuse java streamer almost without changes.

A fact: C# schema more than twice bigger than the java's.

Recently, we have started looking into a task of creating an interactive parser. A generic one.

Yes, we know there are plenty of them all around, however the goals we have defined made us to construct the new implementation.

The goals:

• Parser must be incremental.
  You should direct what to parse, and when to stop.
  This virtually demands rather "pull" than conventional "push" implementation.
• Parser must be able to synchronize a tree with text.
  Whenever the underlying text is changed, a limited part of a tree should to be updated.
• Parser should be able to recover from errors, and continue parsing.
• Parser should be manageable.
  This is a goal of every program, really.
• Parser must be fast.
• A low memory footprint is desired.

What's implemented (VS2008, C#) and put at SourceForge, is called an Incremental Parser.

These are parser's insights:

• Bookmarks are objects to track text points. We use a binary tree (see Bare binary tree algorithms) to adjust positions of bookmarks when text is changed.
• Ranges define parsed tree elements. Each range is defined by two bookmarks, and a grammar annotation.
• There are grammar primitives, which can be composed into a grammar graph.
• A grammar graph along with ranges form a state machine.
• Grammar chains are cached, turning parsing into a series of probes of literal tokens and transitions between grammar chains. This caching is done on demand, which results in warming-up effect.
• Parser itself includes a random access tokenizer, and a queue of ranges pending to be parsed.
• Parsing is conducted as a cycle of pulling and parsing of pending ranges.
• Whenever text is changed a closest range is queued for the reparsing.
• A balance between amount of parsing and memory consumption can be achieved through a detalization of grammar annotation for a range. An active text fragment can be fully annotated, while for other text parts a coarse range can be stored.

We have defined xpath like grammar to test our ideas. See printed parsed trees to get understanding of what information can be seen from ranges.

We have created Java Xml Object Model purely for purposes of our project. In fact jxom at present has siblings: xml models for sql dialects. There are also different APIs like name normalizations, refactorings, compile time evaluation.

It turns out that jxom is also good enough for other developers.

The drawback of jxom, however, is rather complex xml schema. It takes time to understand it. To simplify things we have created (and planning to create more) a couple of examples allowing to feel how jxom xml looks like.

The latest version can be loaded from jxom.zip

We would be pleased to see more comments on the subject.

Although in last our projects we're using more Java and XSLT, we always compare Java and .NET features. It's not a secret that in most applications we may find cache solutions used to improve performance. Unlike .NET providing a robust cache solution Java doesn't provide anything standard. Of course Java's adept may find a lot of caching frameworks or just to say: "use HashMap (ArrayList etc.) instead", but this is not the same.

Think about options for Java:
1. Caching frameworks (caching systems). Yes, they do their work. Do it perfectly. Some of them are brought to the state of the art, but there are drawbacks. The crucial one is that for simple data caching one should use a whole framework. This option requires too many efforts to solve a simple problem.

2. Collection classes (HashMap, ArrayList etc.) for caching data. This is very straightforward solution, and very productive. Everyone knows these classes, nothing to configure. One should declare an instance of such class, take care of data access synchronization and everything starts working immediately. An admirable caching solution but for "toy applications", since it solves one problem and introduces another one. If an application works for hours and there are a lot of data to cache, the amount of data grows only and never reduces, so this is the reason why such caching is very quickly surrounded with all sort of rules that somehow reduce its size at run-time. The solution very quickly lost its shine and become not portable, but it's still applicable for some applications.

3. Using Java reference objects for caching data. The most appropriate for cache solution is a java.util.WeekHashMap class. WeakHashMap works exactly like a hash table but uses weak references internally. In practice, entries in the WeakHashMap are reclaimed at any time if they are not refered outside of map. This caching strategy depends on GC's whims and is not entirely reliable, may increase a number of cache misses.

We've decided to create our simple cache with sliding expiration of data.

One may create many cache instances but there is only one global service that tracks expired objects among these instances:

private Cache<String, Object> cache = new Cache<String, Object>();

There is a constructor that specifies an expiration interval in milliseconds for all cached objects:

private Cache<String, Object> cache = new Cache<String, Object>(15 * 60 * 1000)

Access is similar to HashMap:

instance = cache.get("key"); and cache.put("key", instance);

That's all one should know to start use it. Click here to download the Java source of this class. Feel free to use it in your applications.

Recently I've proposed to add two new atomic types tuple and map to the xpath/xslt/xquery type system (see "Tuples an maps"). Later I've implemented tuple and map pure xslt approximation. Now I want to present java implementation for Saxon.

I've created TupleValue and MapValue atomic types, and Collections class exposing extension functions api. It's easy to use this api. I'll repeat an example that I was showing earlier:

<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:f="http://www.nesterovsky-bros.com/xslt/functions/public"
  xmlns:p="http://www.nesterovsky-bros.com/xslt/functions/private"
  xmlns:c="java:com.nesterovskyBros.saxon.Functions"
  exclude-result-prefixes="xs f p c">

<xsl:template match="/">
   <root>
     <xsl:variable name="tuples" as="item()*" select="
       for $i in 1 to 20
         return c:tuple(1 to $i)"/>

    <total-items>
      <xsl:sequence select="
        sum
        (
          for $tuple in $tuples return
            count(c:tuple-items($tuple))
        )"/>
    </total-items>
    <tuples-size>
      <xsl:sequence select="count($tuples)"/>
    </tuples-size>
    <sums-per-tuples>
      <xsl:for-each select="$tuples">
        <xsl:variable name="index" as="xs:integer" select="position()"/>

        <sum index="{$index}" value="{sum(c:tuple-items(.))}"/>
      </xsl:for-each>
    </sums-per-tuples>

    <xsl:variable name="cities" as="element()*">
      <city name="Jerusalem" country="Israel"/>
      <city name="London" country="Great Britain"/>
      <city name="Paris" country="France"/>
      <city name="New York" country="USA"/>
      <city name="Moscow" country="Russia"/>
      <city name="Tel Aviv" country="Israel"/>
      <city name="St. Petersburg" country="Russia"/>
     </xsl:variable>

    <xsl:variable name="map" as="item()" select="
      c:map
      (
        for $city in $cities return
        (
          $city/string(@country),
          $city
        )
      )"/>

    <xsl:for-each select="c:map-keys($map)">
      <xsl:variable name="key" as="xs:string" select="."/>

      <country name="{$key}">
        <xsl:sequence select="c:map-value($map, $key)"/>
     </country>
    </xsl:for-each>
  </root>
</xsl:template>

</xsl:stylesheet>

Download java source.

P.S. I would wish this api be integrated into Saxon, as at present java extension functions are called through reflection.