Convert word2vec bin file to text
I use this code to load binary model, then save the model to text file,
from gensim.models.keyedvectors import KeyedVectorsmodel = KeyedVectors.load_word2vec_format('path/to/GoogleNews-vectors-negative300.bin', binary=True)model.save_word2vec_format('path/to/GoogleNews-vectors-negative300.txt', binary=False)
Note:
Above code is for new version of gensim. For previous version, I used this code:
from gensim.models import word2vecmodel = word2vec.Word2Vec.load_word2vec_format('path/to/GoogleNews-vectors-negative300.bin', binary=True)model.save_word2vec_format('path/to/GoogleNews-vectors-negative300.txt', binary=False)
On the word2vec-toolkit mailing list Thomas Mensink has provided an answer in the form of a small C program that will convert a .bin file to text. This is a modification of the distance.c file. I replaced the original distance.c with Thomas's code below and rebuilt word2vec (make clean; make), and renamed the compiled distance to readbin. Then ./readbin vector.bin
will create a text version of vector.bin.
// Copyright 2013 Google Inc. All Rights Reserved.//// Licensed under the Apache License, Version 2.0 (the "License");// you may not use this file except in compliance with the License.// You may obtain a copy of the License at//// http://www.apache.org/licenses/LICENSE-2.0//// Unless required by applicable law or agreed to in writing, software// distributed under the License is distributed on an "AS IS" BASIS,// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.// See the License for the specific language governing permissions and// limitations under the License.#include <stdio.h>#include <string.h>#include <math.h>#include <malloc.h>const long long max_size = 2000; // max length of stringsconst long long N = 40; // number of closest words that will be shownconst long long max_w = 50; // max length of vocabulary entriesint main(int argc, char **argv) { FILE *f; char file_name[max_size]; float len; long long words, size, a, b; char ch; float *M; char *vocab; if (argc < 2) { printf("Usage: ./distance <FILE>\nwhere FILE contains word projections in the BINARY FORMAT\n"); return 0; } strcpy(file_name, argv[1]); f = fopen(file_name, "rb"); if (f == NULL) { printf("Input file not found\n"); return -1; } fscanf(f, "%lld", &words); fscanf(f, "%lld", &size); vocab = (char *)malloc((long long)words * max_w * sizeof(char)); M = (float *)malloc((long long)words * (long long)size * sizeof(float)); if (M == NULL) { printf("Cannot allocate memory: %lld MB %lld %lld\n", (long long)words * size * sizeof(float) / 1048576, words, size); return -1; } for (b = 0; b < words; b++) { fscanf(f, "%s%c", &vocab[b * max_w], &ch); for (a = 0; a < size; a++) fread(&M[a + b * size], sizeof(float), 1, f); len = 0; for (a = 0; a < size; a++) len += M[a + b * size] * M[a + b * size]; len = sqrt(len); for (a = 0; a < size; a++) M[a + b * size] /= len; } fclose(f); //Code added by Thomas Mensink //output the vectors of the binary format in text printf("%lld %lld #File: %s\n",words,size,file_name); for (a = 0; a < words; a++){ printf("%s ",&vocab[a * max_w]); for (b = 0; b< size; b++){ printf("%f ",M[a*size + b]); } printf("\b\b\n"); } return 0;}
I removed the "\b\b" from the printf
.
By the way, the resulting text file still contained the text word and some unnecessary whitespace which I did not want for some numerical calculations. I removed the initial text column and the trailing blank from each line with bash commands.
cut --complement -d ' ' -f 1 GoogleNews-vectors-negative300.txt > GoogleNews-vectors-negative300_tuples-only.txtsed 's/ $//' GoogleNews-vectors-negative300_tuples-only.txt
the format is IEEE 754 single-precision binary floating-point format: binary32http://en.wikipedia.org/wiki/Single-precision_floating-point_formatThey use little-endian.
Let do an example:
- First line is string format: "3000000 300\n" (vocabSize &vecSize, getByte till byte=='\n')
Next line include the vocabword first, and then (300*4 byte of float value, 4 byte for eachdimension):
getByte till byte==32 (space). (60 47 115 62 32 => <\s>[space])
then each next 4 byte will represent one float number
next 4 byte: 0 0 -108 58 => 0.001129150390625.
You can check the wikipedia link to see how, let me do this one as example:
(little-endian -> reverse order) 00111010 10010100 00000000 00000000
- first is sign bit => sign = 1 (else = -1)
- next 8 bits => 117 => exp = 2^(117-127)
- next 23 bits => pre = 0*2^(-1) + 0*2^(-2) + 1*2^(-3) + 1*2^(-5)
value = sign * exp * pre