[Architecture] The State of our Branches
antranig.basman at colorado.edu
Wed Nov 5 08:42:19 EST 2014
Kaspar and I have been at work for some months on significant
improvements to our core architecture - unfortunately during this time
it has not been possible to process new contributions. This set of
long-term branches are now getting very close to being merged so I
thought I would provide some notes regarding the changes and also some
details about the merging endgame, timeline and last requirements.
My work has mostly been under the headline of
http://issues.gpii.net/browse/GPII-434 - which in turn was revealed as a
requirement some months ago in order to be able to merge Simon Bates'
work for GPII-94 enabling us to upgrade to current versions of node.js
on Windows. As Steve Lee will attest, our current set of version
requirements on Windows are sufficiently behind the curve as to make
installing the GPII on Windows quite intricate and error-prone. This
work also enables the use of "Integration Tests" - the configuration
previously applied as "Acceptance Tests" in the architecture-specific
branches has been hoisted up into the universal project, allowing us to
exercise the complete workflow of the GPII system for these scenarios in
a portable way, allowing us to resolve any issues that they raise
separately from the action of the settings handlers and solution
launchers on the real systems.
Kaspar's work is under the headline of
http://issues.gpii.net/browse/GPII-941 - this makes improvements to our
MatchMaker payloads that will make MatchMaker integration significantly
easier in future (previous integrations have taken wholesale forks of
universal) as well as preparing for the work required for the
Context-Aware Server and Environmental Reporter. This work also makes
essential improvements to the Ontology Server which will be necessary,
for example, to enable security profile settings in the PMT as well as
simplifying MatchMaker implementation.
My branches for this work are at
with a corresponding Windows branch at
These are now at the stage where they function "roughly as well" as the
old system but unfortunately the improved settings handler and lifecycle
manager have shed some new light on the exact causes of our test
failures on Windows which prompted this work in the first place.
Kaspar's branch is at
https://github.com/GPII/universal/pull/279/files which has been merged
up with my branch as of about a week ago.
The new workflow enables the deployment of settings handlers which are
i) asynchronous, and ii) which the system will automatically retry until
the required result is received. Comparing the results of successive
settings snapshots on Windows has revealed a very peculiar effect, which
I can only explain in terms of a cross-linkage between the effects of
different settings handlers on Windows.
PiratePad seems down at the moment but a sample transcript can hopefully
seen in future at
This shows the Windows cursor settings apparently snapshotted in the
middle of what must be an OS-initiated process to reset their values.
Three of them hold their original values and the other 10 have been
reset. On the following snapshot, all 13 have been reset. My belief is
that this process is initiated asynchronously by the OS in response to a
desktop theme change - which itself is a change of which we have no
means currently of detecting the time point of application (see GPII-581).
So a little further work is required in order to produce acceptance
tests which pass "essentially all the time" - I will need to improve
this branch so that read retries also involve write retries. However I
think this implementation is close enough to the final one that we
should start coordinating the final phases of the merging process.
Some important changes in the GPII-434 branches:
i) As revealed by GPII-1001, many of our settings handlers had faulty
implementations for the "get" cycle, as well as in general having a
faulty or absent idiom for the "removal" of values that had previously
been found to be unset when the system started up. Although it is
architecturally dubious, the quickest way of resolving this issue
without completely disrupting our settings handler contract has been to
handler payload value. When re-presented with this value, the "set"
cycle will interpret this as an instruction to entirely remove the
This means that our settings handler payloads have become invalid when
considered as JSON values. The sounder solution in the long term is to
move our settings handler API over towards the form operated in
Infusion's ChangeApplier API -
as well as allowing us to account for value deletion in a stable way, it
will also allow in time for encoding other important operations such as
insertion of elements into an array. In the sort term, if we find the
requirement for transmitting or storing these payloads, they will need
to go through a separate phase of escaping and de-escaping in order to
account for their status as improper JSON. We may imminently find
ourselves doing this in upcoming work on what could be called the
"WebSockets (inverse) Settings Handler" for communicating with GPII
browser plugins/extensions in the role of a settings handler.
Javi - in the meantime, for GPII-1001, you can adopt the workflow in the
gpii.setttingsHandlers.setSettings method at
- this uses an Infusion shorthand for accounting for the difference
between settings writing and deletion
ii) Wholesale removal of the "LifecycleManagerServer" which has been
found to be a redundant architectural component
iii) As mentioned before, the LifecycleManager has a fully asynchronous
workflow based on "micropromises" (introduced in FLUID-5513), and all
test cases previously taking the form of "acceptance tests" are now
available as "integration tests" (runnable via IntegrationTests.js) at
This branch still contains a moderate amount of debugging code but I
think it is stable enough for Kaspar to merge up his branch with again.
In particular there will be changes to the LifecycleManager which will
cause conflicts - I've moved this over to an invoker-based model since
it would be too awkward to inject in the "retrying configuration"
otherwise. Also the removal of the "LifecycleManagerServer" will cause
conflicts with your current work on fixing the "update" action for the
new MatchMaker semantic.
iv) Others (e.g. the new standalone drivers named "gpii.js", the new
module resolution system that eliminates hard-coded file references
between modules, improvements to grunt scripts and linting for all
projects, the "nameResolver" system for locating mocks for settings
handlers and actions, etc.)
Colin is at work reviewing the underlying pull requests to Infusion and
Kettle that all this work depends on - given the large-scale nature of
the changes to GPII I think it makes sense for us to adopt a process of
"collective review" for the core architecture changes since the branches
contain work from so many different contributors.
I and Javi still need to produce a corresponding branch of the Linux
repository which syncs up with this work - this depends on having
GPII-1001 resolved first - the other changes in packaging (e.g. removal
of acceptance tests) should be relatively straightforward. There will
also need to be a branch of the android repo too.
More about this at arch meeting later -
More information about the Architecture