NOTE: The SVN repository isn''t up to date with these changes. It will
be sometime over the weekend.
Thanks & happy reading,
-John
286238: (dinov)
Currently ngen will try and produce an image that?s probably around 1gig in size
(or larger) for Microsoft.Scripting.dll due to the reflected caller additions.
This cuts back the number of FastCreate?s we?ll do and gets this back to a
reasonable level. It also adds a cache for non-dynamicmethod?s so that we only
create one per method. This should help w/ both the throughput regressions as
well as the fact that I broke ngen.
Also included is the fix for the out param reported on the mailing list
yesterday.
286963: (tomat) Ruby bug fixes
Fixes the following Ruby bugs (no major change in design needed).
ID Assigned To Title
304051 Tomas Matousek ruby: explicitly set $! in the ensure clause
should not be preserved
303812 Tomas Matousek ruby: $! is not preserved in the ensure
clause when the exception is thrown in the else clause
303774 Tomas Matousek ruby: redo should not re-evaluate the loop
condition
303703 Tomas Matousek ruby: "break" does not break out
from the lambda call
298222 Tomas Matousek ruby: explicitly setting $! in the rescue
exception parameter does not preserve $!
297959 Tomas Matousek ruby: $! in the ensure clause should not be
nil when an exception is raised, but no matched rescue clause is found
293040: (dinov) Preparation to fix infinite recursion issue
These are the final set of small fixes which are necessary before we can fix the
infinite recursion issue.
1. Switch from using Type.Missing to our own MissingParameter type. We
can''t pass Type.Missing through Reflection.Invoke calls.
2. Fix Python''s DoOperationBinderHelper to avoid casts which
box/unbox values (this is just due to an identity of integer test in test_copy)
3. Add a missing conversion in Ruby
4. Check IsInstanceOfType in Cast to catch COM objects interface tests
correctly
5. Makes evaluation of addresses work the same as emitting - we can end up
evaluating an expression before getting the address and that expression can
throw.
6. Fix an issue w/ return types on properties not going through the
binder''s converter
7. Add support for strongly typed delegates in the eval case
8. Fixed an issue w/ rethrowing exceptions in evaluate mode
9. Added support for returning from a VoidExpression
10. Fix an issue w/ arrays in evaluate mode where we try and cast to object[]
when we have an array of byte[]
Also fixing this bug:
304475 dir(System.Reflection.Emit.AssemblyBuilder) throw TypeError
Where we just need to check for null in DynamicMixin.
293085: (tomat) Fixes RubyForge bug #14425.
Ruby allows jump statements (return, redo, retry, break and next) to appear in
Boolean expressions as right operands.
Examples:
def foo x
x and return ?true?
x or return ?false?
?Unreachable?
end
foo true # ?true?
foo false # ?false?
This shelveset changes the grammar to support this feature and adds AST node
ConditionalJumpExpression that represents <condition> and/or
<jump-statement> expression. Also adds a unit test of the node.
293153: (jomes) generics: implements support for constructing generic types
Pretty simple, implements support for creating generic .NET types like
List<T>, Dictionary<K,V>, etc.
The syntax in Ruby is something like this:
require "mscorlib"
IntList = System::Collections::Generic::List.of(System::Int32)
a = IntList.new
a.add 1
a.add 2
a.add 3
assert_equal(a.count, 3)
I also added support for the [] syntax that IronPython uses:
System::Collections::Generic::Dictionary[String, String]
293329: (mmaly) Bye bye, WeakCondition
This change gets rid of the WeakCondition and updates all former calls to that
to generate proper casts. Ruby has factory it wanted which generates the casts
and up-casts to object. The cast to bool must be dynamic so that the helper gets
called. Alternative would be calling the helper explicitly or doing something
yet smarter.
There is one temporary change in the ReferenceArgBuilder.cs The builder
generates code:
ElementType tmp;
arg is StrongBox<ElementType> ? tmp =
RuntimeHelpers.GetBox<ElementType>((StrongBox<ElementType>)parameter),
tmp : RuntimeHelpers.IncorrectBoxType(StrongBox<ElementType>);
Notice the comma in the first expression. If we cast the whole thing to object,
the currently not quite complete implementation of calling methods with by ref
arguments will do the wrong thing. Temporarily we cast the 2nd expression to
ElementType as it will throw inside the helper anyway and the cast will never
execute. The ideal solution is for the binder to generate a better tree all
around, which will play better with the upcoming ?emit as? implementation
overhaul.
293941: (tomat) Ruby libraries
Prepares built-ins for a split: the types and functionality that the compiler
directly depends on will be kept in Ruby.dll while all Ruby visible methods will
be factored out to Ruby.Libraries.dll. Also adds Ruby.Libraries and
ClassInitGenerator projects to the Rowan solution ? they depend on Ruby.dll and
M.S.dll, so they should be rebuilt each time the dependencies are rebuilt.
293960: (jomes) fixes inheriting from a .NET type
Very simple change, fixes inheriting from a .NET type in Ruby
1. We weren?t handling TypeTrackers
2. We weren?t exposing CLS members on the derived type
294389: (mmaly) Switch statement upgrade
I managed to rewrite the switch statement yesterday. While this change doesn?t
yet add the missing interpreted mode, it removes all Jscript-isms. The switch
statement is true switch. Takes an int and routes to given label. The labels are
marked with _constant_ values. Switch will choose dynamically whether to emit as
jump table or ?if? statements.
If the labels are integer-like constants, it will generate the DLR switch. If
not, it will also generate the DLR switch (cool, huh?) except preceded with an
if statement which ?encodes? the switch labels into a constant integer and then
proceeds to the direct switch with that constant. This eliminates the need for
goto to do the fall through.
In the second (via encoding) case, you?ll see the if ? elseif ? elseif pattern
first, in which the bodies are only (.bound #2) = ? and then switch over the #2
temp.
DLR switch now checks for unique label values, and does it in 3 ways:
? If there are small number of cases (< 10), it uses dumb O(N^2) algorithm
? If there are cases values of which are in reasonable range (max ? min <
1024), it uses BitArray to track the unique values
? Else, it falls back to dictionary
Only the generation into the pure switch is affected. When generating the if
..elif ..elif prefix, the generated label numbers are unique by definition so
that path remains the same. However, for the switch table code path, the code
merges bodies of the case clauses. Essentially each case clause whose label is
not unique is merged with the previous clause?s body. The bodies must stay in
place because of the fall-through, but no one can ever jump to the label of the
non-unique statement so it is safe to merge them with the previous bodies.
294586: (jomes) ruby extension methods on interfaces
Implements extension methods on interfaces for Ruby. What this means it that
.NET types will get Ruby methods injected onto them if they implement certain
interfaces. For example, types that implement IEnumerable will behave just like
a Ruby classes that mix in Enumerable.
I implemented this by treating interfaces as if they were mixed in modules.
These interfaces have extension methods now:
? IEnumerable
? IComparable
? IList<object>
? IDictionary<object, object>
IList<object> and IDictionary<object, object> were factored out of
Array and Hash, respectively. Some investigation remains to see if these can be
made to support all generic types or if they could use IList and IDictionary
instead, but I didn?t attempt that yet.
Detailed change description:
? Added a new attribute RubyExtensionInterfaceAttribute, updated
ClassInitGenerator & Initializer to understand it
? Added the MixinInterfaces property to RubyClassAttribute, which tells
ClassInitGenerator to inject all valid interface methods it finds onto the type
(useful if you?re extending a CLR type and you want everything).
o StringOps uses this feature to get Enumerable & Comparable support.
? RubyExecutionContext keeps track of interface mappings & mixes in
the interface?s RubyModule onto new .NET types
? IEnumerableOps implements each & includes Enumerable
? IComparableOps implements <=> & includes Comparable
? IDictionaryOps takes methods from HashOps (which were already written to
operate on IDictionary<object, object>)
? IListOps takes methods from ArrayOps
o Helpers are needed for AddRange, GetRange, InsertRange, and RemoveRange,
which IList<T> doesn?t support. Also, several methods are different
between ArrayOps and IListOps
294789: (dinov) Infinite recursion fix
This changes rule creation so it only runs the test once and evaluates the
target to avoid the test changing and getting infinite recursion.
This is done by pushing the update & invoke out of
DynamicSite.UpdateBindingAndInvoke and all the way down into the internal
portions of ActionBinder and the rule caching mechanism. We now pass in the
site, the site''s target (by-ref), and the site''s rule list
(by-ref) and let the action binder update these for us. This reduces a bunch of
the duplicated code in DynamicSite.Generated.cs.
Part of the reason to push this in is that the site HAS to have its target
delegate set before executing the target of the rule. If not the site could
recurse on its self, not have the target set, and therefore we limit the amount
of stack that can be used - as well as increase the # of calls which
don''t use the optimized target.
294962: (dinov) Updates ExtensionTypeAttribute
This updates ExtensionTypeAttribute to no longer have any DynamicType
dependencies. It also removes some Python specific pieces to it (support for
deriving from int, etc?) and removes ?using M.S.T;? from everyone using it just
for ExtensionTypeAttribute (this is most of the files changed).
294999: (dinov)
This change implements the helper methods on PropertyTracker and switches
GetMemberBinderHelper over to using these helper methods. This change
ultimately removes dependencies on ReflectedProperty and ReflectedIndexer.
Mostly this is moving the logic in GetMemberBinderHelper over into
PropertyTracker.
Add new ErrorInfo class which is used for reporting information about how an
error should be reported to the user. The error can either be reported as an
exception which gets thrown or a value that gets returned. Added new methods to
ActionBinder which enable languages to produce errors. Eventually all of the
error production will move into this format.
Small tweaks:
Move MakeCallExpression from BinderHelper onto Binder. The only
reason it was on BinderHelper was because it needed to access the Binder to
provide conversions.
Fixed the iteration in MethodBinder over the dictionary (which
is bogus, we have a CodePlex bug on this reported by a user)
Implemented Call on MethodTracker ? this is used when
PropertyTracker falls back for private binding.
Exposed IsStatic directly off of PropertyTracker.
295337: (mmaly) Redesigning the DLR AST Walkers
(Walker6 passed SNAP already, Walker7 contains few cosmetic updates such as
missing comments, removing few redundant argument checks, adding few extra ones
etc. I hope it also passes)
To fix the ?non-empty stack upon entry to try? bug, I am going to need a better
walker than what is easy to implement in our system (one that I can use to
re-generate the AST along the way). The current walker design doesn?t lend
itself easily to that purpose (the Walk methods on the nodes do just that ?
walk, but I?d have to write a type-based switch to implement my walker, rather
than an enum-based switch).
Second motivation is LINQ which doesn?t have walker on its nodes so to get
closer to that model, our walkers must live completely outside of the AST.
This checkin achieves these two goals in a following ways:
1) Each node is given an enum (this enum started with LINQ?s values, some
are commented out as we don?t use them (yet ? CheckedAdd for example), some are
added (statements). The UnaryOperator and BinaryOperator enums got swallowed by
the big enum which nicely plays that role.
2) The walker is combination of generated and hand-written code and the
Walk methods from the AST nodes are gone. The individual walkers haven?t changed
except to call WalkNode(node) instead of node.Walk(this) to perform the node
walk. The WalkNode will switch based on the node type (enum), direct cast and
perform walk identical to that of the original node?s walk.
What I like about having the walker completely outside of the tree is that the
AST nodes don?t dictate the order of walking or don?t specify one way as
?primary?. This way any walker (and we?ll eventually need a reverse walker)
plays by the same rules on the same playing field.
Next steps in this direction:
? Make all leaf AST nodes sealed
? Remove codegen from the AST nodes
? Merge AST node that can be merged (say, after we remove the codegen out,
there is no need to have break and continue as separate AST nodes because
they?ll hold identical information, same for CodeContextExpression and
EnvironmentExpression for example). So we?ll end up with fewer nodes as a
result, which is a goodness.
295459: (dinov)
This change is just re-organizing DynamicHelpers and RuntimeHelpers in
preparation for the removal of DynamicType. Previously these two classes had no
real strong distinction. Why does something go into DynamicHelpers? Why does
it go into RuntimeHelpers? Who knows. There is now one distinction: Things in
DynamicHelpers are going away, things in RuntimeHelpers are staying.
DynamicHelpers its self moves into M.S.Types indicating that it will be removed
real soon now (I don?t actually move the file, it?ll be gone in a day or two).
What this is ultimately going to accomplish is that all of the IronPython type
system will be able to come up in one chunk w/ hopefully no code changes. From
there it?ll be a few renames away from being integrated into the IronPython code
base. Because there are a whole lot more references to
DynamicHelpers.GetDynamicType* then there are to the other functionality in
DynamicHelpers it seems natural to keep DynamicHelpers as the home of
GetDynamicType and friends.
So most of this change is just moving things from DH to RH and adding ?using
M.S.Types;? to IronPython. There?s one additional tweak in that I remove
LanguageContext.DeleteMember. JS was using DynamicType for its implementation
and the only real caller was IronPython. This gets replaced by a
PythonOps.DeleteAttr which closely mimics the TryGetAttr method we already have.
Also some code only used by Python (CheckTypeVersion) gets pulled up into
PythonOps and a couple of places where we stop getting type names from
DynamicType.
295467: (jomes) inject onto IList vs. IList<object>
Pretty simple, just changes the IList<object> member injectors to inject
onto IList instead, because a lot more things implement IList (especially
notable are ArrayList and List<T> for any type T).
I needed a couple of minor bug fixes?the interface injection was injecting
things onto non-concrete generic types. I also changed the generator back to
using a sorted dictionary so Initializer.Generated.cs doesn?t change so much
each time (which you can see if you look at the diff, some stuff changed that
shouldn?t have).
295500: (dinov)
This starts exposing MethodGroup?s instead of BuiltinFunction?s.
First off one ?big? change is FastCallable is getting moved into M.S.Types
(which will get pulled up). This also causes a bunch of files to get using
M.S.Types for references to FastCallable. CallType remains in
Microsoft.Scripting as it?s a component of the MethodBinder.
This change alters some MemberTracker setup. I?ve made MemberGroup?s no longer
a MemberTracker. Instead MemberGroup?s are just ways that we talk about groups
of members. I?ve added a MethodGroup which is a collection of methods.
MethodGroup?s are also unique per actual .NET method (and generic instantiation
for generic methods). That allows us to use a simple comparison on them to
check for identity (this is enforced by the ReflectionCache).
From here it?s basically pretty straight-forward: The DLR gets MemberGroup?s,
for method binding it creates MethodGroup?s (once we?re resolved to just a bunch
of methods). All languages except for Python currently expose MethodGroup?s
directly to the user. Python transforms these into BuiltinFunction?s.
There?s still more work to be done: This change doesn?t remove
CallBinderHelper?s recognition of BuiltinFunction/BuiltinMethodDescriptor?s. We
should also more strictly base Python?s identity of BuiltinFunction?s on
MethodGroup?s and then we can also use simple object identity for tests against
BuiltinFunction?s ? currently Python is sharing the hash-key w/ ReflectionCache
which isn?t necessary. Finally we don?t have perfect one-to-one mappings
between methods and MethodTracker?s. This is because we use the MethodInfo as
the key which can be duplicated based upon from what type you got the
MethodInfo. But this change is already big enough that it seemed prudent to
wait on those next steps.
296822: (jomes) Ruby Class Variables
Implements support for Ruby class variables (aka static fields)
Mostly, this is straightforward. We add a dictionary to RubyModule for storing
the values, and add methods for getting and setting the variables. The class
variable AST node turns into get/set calls. Ruby resolves class variables in
method resolution order, so it searches all base classes & mixins. Setting
variables is interesting?the MRO is searched, and if the variable is found it?s
modified, otherwise, it?s added to the current Module/Class.
A few AST changes are needed?unlike instance variables (which are resolved on
the current ?self?), class variables are lexically bound to the enclosing
module/class. So we need scoping for modules, and methods need the ability to be
closures. We close over the module?s ?self?, but because the DLR merges
variables by name, we need to generate a new name for each module. Also, DLR
closures don?t work unless you flow in the correct CodeContext, so
RubyMethodInfo?s need to hold on to their CodeContext, and flow in the right
one.
297031: (jomes) ruby monkey patching of .NET types
A few minor fixes to allow Ruby to add methods to .NET types, aka ?monkey
patch?. The reason this wasn?t working is that Ruby exposes .NET namespaces as
NamespaceTrackers & .NET types as TypeTrackers. We were already converting
TypeTrackers to RubyClass under certain circumstances; now we also convert
NamespaceTrackers to RubyModules. For this to work, a RubyModule can have a
NamespaceTracker backing it. When we go to lookup a constant on the module, we
check the module?s constants, and then fall back to the namespace.
This design is not optimal... we need to come up with a better way for
language?s type objects to interoperate. But it?s intended to unblock some of
our .NET interop scenarios in the short term.
297060: (jomes) fix scoping of constant variables
Ruby constants are supposed to bind lexically to the enclosing class/module, but
this wasn?t working. Pretty simple change, ConstantVariable now binds to the
lexically enclosing module, or Object if an enclosing module isn?t present.
There was already a test for this, but it wasn?t running against IronRuby, so I
enabled it & added a few more test cases.
297066: (jomes) Ruby instance_eval/class_eval/module_eval support (for the
non-string overloads)
Implements basic support for the instance_eval, class_eval, module_eval
overloads that take a block (not the ones that take strings, that requires full
?eval? support). This is pretty easy to do, because we just create a new Proc
with a different ?self? and call the block.
instance_eval on a Module/Class seems to work a bit differently, so I?m throwing
NotImplemented until we can make the necessary infrastructure changes to handle
it correctly.
297179: (dinov)
Next change in the line of changes to remove DynamicType. This one removes
ReflectedEvent/BoundEvent from the parlance of the DLR:
ReflectionCache.GetReflectedEvent moves to DynamicTypeOps and callers are
updated.
Python gets a PythonSetMemberBinderHelper for dealing w/ the assignment logic
back to a ReflectedEvent.
We now pass the type we?re doing the lookup from for ReturnMemberTracker (this
is our version of the .NET ReflectedType property which shows where each member
came from)
The DLR currently generates calls to static helper methods for doing the
addition/subtraction to the event object.
Fixed an issue where getting a TypeGroup?s DeclaringType/Name could throw
And finally manifest the in-place add/subtract methods onto EventTracker?s and
BoundEventTrackers (these aren?t directly defined to deal w/ typing issues where
we need the delegate to be strongly typed).
2975790: (jflam) Enables support for Rubinius spec suite
Adds Rubinius spec suite for Array, Hash and String. Dir is also added, but
tests are not enabled for this yet.
Adds a modified version of the Rubinius spec runner that uses only features
implemented by IronRuby. These are the mini_mock, mini_rspec, mspec,
mspec_helper, rspec_helper, simple_mock, and spec_helper files in \Tests.
Adds/fixes features to IronRuby to support running the test suite:
- Fixes require behavior in loader.cs to allow files to be loaded more than
once
- Fixes __FILE__
- Adds File.dirname() method
- Adds RubyExtensionModuleAttribute and marks IListOps, IComparableOps,
IDictionaryOps, IEnumerableOps using this attribute.
- Adds implementation of const_defined?, alias_method and remove_method
Adds a new utility (IronRuby.Libraries.Scanner) that generates a YAML file that
contains a list of all implemented singleton and instance methods (note that
this is written in C# 3.0).
297638: (mmaly) Removing callwiththis
I?ve had CallWithThisExpression in my sight for a while now and as Nandan and
Jitu made progress on the binder, it is no longer used so it can go now.
Additionally, MSAst.Arg was used only as a temporary information storage and was
never actually part of the AST (CallSignature would throw it away) so I got rid
of it too. In Python it meant even cleaner code.
297654: (jomes) splat and send
This change enables splatting of argument lists when calling Ruby library
methods that are implemented in C#. Before this change, splatting only worked if
you?re calling methods defined in Ruby. Fixing this is pretty straightforward: I
just flatten out any splatted lists before passing them to the MethodBinder.
(The method binder expects things flattened down to Types?it doesn?t use
ArgumentKinds).
I also implement Ruby?s send and __send__ methods, which are used by a lot of
tests that John Lam wants to enable. The implementations are very simple &
slow: they just create a new site with the correct InvokeMemberAction and Invoke
it. Not optimal, but it should be okay until we integrate send directly into the
method binder (Tomas is already planning on refactoring the method binder).
297839: (tomat) Ruby Singletons and Method Lookup
Adds support for singleton classes and fixes method lookup. A singleton class is
a class that adds additional members to object instances. In the following
example, ?class << x? defines a singleton class for instance x.
x = Object.new
class << x
def foo
end
end
x.foo
Object.new.foo # error
Singleton class is a first class object, you can store it to a variable and even
create another singleton class for it. The class itself is also defining some
methods. Those are implemented as extension methods in SingletonOps.
There are many peculiarities involved. I?ve also found at least 2 bugs in MRI
during my investigation.
I?ll write a spec and present the internals of singletons and method lookup (as
I understand them) on Friday?s IronRuby meeting.
The shelveset also implementation rescue expression, rescue statement and symbol
constructors, which are handy for testing.
297943: (dinov) Remove references to built-in functions
This change removes the remaining references to BuiltinFunction?s (and friends).
Instead these objects now implement IDynamicObject and use the CallBinderHelper
to produce a call rule.
Supporting changes:
1. Needed to split RuntimeHelpers into 2 parts (RuntimeHelpers and
BinderOps). This is due to a FxCop error where RuntimeHelpers is getting too
complex and including things from too many classes. BinderOps are just the
methods that are called from generated code to do binding (calls, create
instance, get member, etc?)
2. Moved StrongBox helper functions to CompilerHelpers as these need to be
accessed outside the binder.
3. Added support for passing isBinaryOperator/isReversedOperator on
CallBinderHelper as well as ability to set Instance. The hope here is that
these flags go away from the MethodBinder (they?re awfully Python specific) and
that we have a more consumable way to do call binding in the future but this is
a reasonable intermediate step. The CallBinderHelper also no longer stores the
instance type instead getting it from the _instance expression.
297991: (dinov)
This moves the Microsoft.Scripting.Types namespace to IronPython. No namespaces
are renamed during this remove (that touches over 100 files, so it?s next).
Appropriate resource strings also get moved to IronPython, and much of the
DynamicType machinery is marked as internal.
This reduces a release build of MS.Scripting to 868,352 bytes from 958,464
saving 90,112 bytes or 9.4%.
This reduces a debug build of MS.Scripting to 995,328 bytes from 1,118,208
saving 122,880 bytes or 10.9%.
298014: (jomes) Case Expressions in Ruby
This change implements case expressions in Ruby. Case expressions look like
this:
case x
when a0, ? aN, *splat0: <body0>
when b0, ? bN, *splat1: <body1>
?
else: <bodyN>
end
It translates roughly into:
if (a0 === x || ? || aN === x || <testarray>) {
body0
} else if ( ? ) {
body1
} else {
bodyN
}
Where <testarray> is:
result = false;
foreach (object obj in splat0) {
if (obj === x) { result = true; break; }
}
And of course, === is Ruby?s case equality method ?
It?s pretty straightforward?the only tricky part is cracking open the splatted
array, if present.