Prelude.undefined while unpickling

[ftomassetti]

Hi, I am running this code on a protocol 2 file:

main :: IO ()
main = do putStrLn "Start"
          byteString <- S.readFile worldFileName :: IO S.ByteString
          let res = unpickle byteString
          case res of
               Left err -> putStrLn $ "Can't unpickle .\nUnpickling error:\n " ++ err
               Right v -> putStrLn "Well done!"
          putStrLn "Done"

And I keep getting:

Start
civs: Prelude.undefined

where civs is the name of my app. Do you know what I am doing wrong?

[noteed]

Probably nothing, I think I didn't implement everything. I will get a look. Could you please give me a minimal test file that triggers the problem ?

(I'm very happy to get this issue; I didn't know someone was using this library !)

[noteed]

Indeed there are plenty of undefined in the code (i.e. corresponding to non-implemented op-codes). I'm willing to add missing op-codes for "standard" pickle data (i.e. probably no support for user-defined classes and instance, or for things supported by pickle but not by cPickle).

[ftomassetti]

Thank you for writing and sharing this library. I am a real beginner in Haskell but I would try to help if I could find the documentation for the pickle protocol

[noteed]

Have a look at the README, I think there is some pointer to learn the Pickle format. That being said if you can give me a sample file that you're trying to parse (possible the smallest example that triggers the undefined), I would look into fixing the problem.

[ftomassetti]

The file is quite large, but looking at it with pickle tools these are the opcodes used at the beginning:

    0: \x80 PROTO      2
    2: c    GLOBAL     'geo World'
   13: q    BINPUT     0
   15: )    EMPTY_TUPLE
   16: \x81 NEWOBJ
   17: q    BINPUT     1
   19: }    EMPTY_DICT
   20: q    BINPUT     2
   22: (    MARK
   23: U        SHORT_BINSTRING 'biome'
   30: q        BINPUT     3
   32: ]        EMPTY_LIST
   33: q        BINPUT     4
   35: (        MARK
   36: ]            EMPTY_LIST
   37: q            BINPUT     5
   39: (            MARK
   40: U                SHORT_BINSTRING 'ocean'
   47: q                BINPUT     6
   49: h                BINGET     6

After that we have thousands of BINGET, BINFLOAT, BININT1

The end of the file looks like:

17212214: G                    BINFLOAT   0.2535521984100342
17212223: G                    BINFLOAT   0.2510862112045288
17212232: G                    BINFLOAT   0.25246264934539797
17212241: e                    APPENDS    (MARK at 17207632)
17212242: e                APPENDS    (MARK at 14848849)
17212243: u            SETITEMS   (MARK at 14848742)
17212244: u        SETITEMS   (MARK at 22)
17212245: b    BUILD
17212246: .    STOP

[noteed]

Thanks! That's a good start. Looking at the source, GLOBAL is not implemented in the parser, and GLOBAL, NEWOBJ and BUILD are not implemented in the stack machine.

Unfortunately we're in the realm of Python classes and objects:

GLOBAL geo World, EMPTY_TUPLE, NEWOBJ

can be understood as geo.World(), i.e. instanciation of a geo.World class. This is clearly outside the scope of this library. Then a big dictionary is constructed (everything starting at EMPTY_DICT). That is perfectly ok for this library. Then BUILD is called to update the geo.World() instance's __dict__ with the constructed dictionary. Again something that is out of scope for this library.

Now we could decide that having a minimal representation (and not a full blown representation of Python classes and objects) for that case is doable, e.g. augment the Value data type to have

| Instance Text Text (Map Value Value) -- ^ module name, class name, dictionary

I think I'm ok with such an option.

The good news is I think you have a perfectly good workaround: remove that part

2: c    GLOBAL     'geo World'
13: q    BINPUT     0
15: )    EMPTY_TUPLE
16: \x81 NEWOBJ
17: q    BINPUT     1

and also

17212245: b    BUILD

and you should be able to parse that file and obtain a dictionary.

EDIT: actually BINGET must be first implemented but it seems straightforward. I will see to do it.

[noteed]

I have implented BINGET in commit 8f403de.