Zstd installation


#1

Hey there,

I’m having a ton of trouble with getting the zstd Python extension to compile. Win7-64, Python 3.6, Visual Studio build tools 2017 and Windows SDK 8.1.

I haven’t really had this much difficulty before, most packages are a pip away. I feel like I’ve resolved all the dependencies, the error it complains about now are actual syntax errors in hist.c which I’m not equipped to fix. The other modules of zstd compile fine, it’s really just hist, which I don’t even need.
Pretty sure it’s something specific with my config since other folks seem to compile just fine.

hist.c
zstd/lib/compress/hist.c(52): error C2143: syntax error: missing ';' before


zstd/lib/compress/hist.c(52): error C2065: ‘ip’: undeclared identifier
zstd/lib/compress/hist.c(52): error C2100: illegal indirection
zstd/lib/compress/hist.c(52): error C2146: syntax error: missing ‘)’ before
identifier ‘BYTE’
zstd/lib/compress/hist.c(52): error C2059: syntax error: ‘)’
zstd/lib/compress/hist.c(52): error C2297: '
’: illegal, right operand has t
ype ‘const void
zstd/lib/compress/hist.c(53): error C2086: ‘const int BYTE’: redefinition
zstd/lib/compress/hist.c(52): note: see declaration of ‘BYTE’
zstd/lib/compress/hist.c(53): error C2143: syntax error: missing ‘;’ before
'

zstd/lib/compress/hist.c(53): error C2059: syntax error: ‘const’
zstd/lib/compress/hist.c(60): error C2065: ‘ip’: undeclared identifier
zstd/lib/compress/hist.c(60): error C2065: ‘end’: undeclared identifier
zstd/lib/compress/hist.c(62): error C2065: ‘ip’: undeclared identifier
zstd/lib/compress/hist.c(62): error C2100: illegal indirection
zstd/lib/compress/hist.c(68): error C2065: ‘U32’: undeclared identifier
zstd/lib/compress/hist.c(68): error C2146: syntax error: missing ‘;’ before
identifier ‘s’
zstd/lib/compress/hist.c(68): error C2065: ‘s’: undeclared identifier
zstd/lib/compress/hist.c(69): error C2065: ‘s’: undeclared identifier
zstd/lib/compress/hist.c(69): warning C4018: ‘<=’: signed/unsigned mismatch
zstd/lib/compress/hist.c(70): error C2065: ‘s’: undeclared identifier
zstd/lib/compress/hist.c(90): error C2143: syntax error: missing ‘)’ before

zstd/lib/compress/hist.c(90): error C2081: ‘U32’: name in formal parameter l
ist illegal
zstd/lib/compress/hist.c(90): error C2143: syntax error: missing ‘{’ before
'

zstd/lib/compress/hist.c(90): error C2059: syntax error: ‘)’
zstd/lib/compress/hist.c(91): error C2054: expected ‘(’ to follow 'workSpace

zstd/lib/compress/hist.c(172): warning C4013: ‘HIST_count_parallel_wksp’ und
efined; assuming extern returning int
zstd/lib/compress/hist.c(172): error C2065: ‘U32’: undeclared identifier
zstd/lib/compress/hist.c(172): error C2059: syntax error: ‘)’
zstd/lib/compress/hist.c(193): error C2065: ‘U32’: undeclared identifier
zstd/lib/compress/hist.c(193): error C2059: syntax error: ‘)’
error: command ‘C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\B
IN\x86_amd64\cl.exe’ failed with exit status 2

SO! If anyone has run into this, I’d love to hear how you got past it. Otherwise, if someone has a compiled win64 wheel or compiled package of zstd they can share, that would be even better. It would really help me spend my time on ML and not the finer points of windows/python interactions :slight_smile:

Thanks and good hunting!


#2

Try telling VS to compile .c files as C++.


#3

Good call, I was able to compile hist.c using a hand-written compiler call with Tp, qualifying it as a c++ file.

Unfortunately, I have no idea how to tell pip to call the compiler with the same command. Build-ext appears to make its own assumptions about what the files are, and even if I set global /TP flag to treat all files as c++, it still over writes individual files with their own Tc © qualifiers.

I thought I could just let pip do its thing and throw in the hist.obj I compiled manually, but as luck would have it, the --no-clean flag of pip is broken and pip will always clean up the files it generates, meaning I can’t go in steps.


#4

You don’t need zstd if you tell the engine to produce json via --no-compression option. You can do the same when downloading replays from halite. You can also compile zstd.exe separately and decompress everything.


#5

That’s a very good point. I can produce my own training data uncompressed.
I suppose it still limits me if I want to download replays from the site, but I can live without that for now.

Thanks for your help Mr. Turing :wink: