![]() it would be a many-to-many structure like this: ![]() You will need to have a table to keep the relationships between texts and keywords. You can calculate the md5 of the original text or just simply giving a numeric id. However you can keep them also in your keywords array, if you wish. , ! has been eliminated because they do not nothing usually in a keyword-based search. would be something like this: Īnd I'm the next large part! would be: īy the way words I, am, of, I'm, the plus. Maybe some clustering or some performance tricky ways on a good hardware works for you, but if that's not the case, you may find it interesting.įirst, you need an script to just load these 50 billion piece of text one after each other, split them into some words and treat them as a keyword, that means giving them a numeric id and then save them on a table. you have 10 TB of text and you want to load it on your MySQL database and perform a fulltext search on the tables! Okay, it's really challenging! (at least for me!). Output Using GZ Compress = 32.86% Improvement $compressedLength = strlen($compressed) Įcho "Using GZ Compress = ", 100 - number_format(($compressedLength / $base64Length ) * 100, 2), "% of Improvement", PHP_EOL Įcho "Using Deflate = ", 100 - number_format(($deflateLength / $base64Length ) * 100, 2), "% of Improvement", PHP_EOL $encode = base64_encode(gzencode($original, 9)) $deflate = base64_encode(gzdeflate($original, 9)) $compressed = base64_encode(gzcompress($original, 9)) What you should be looking at is compression and i think you should look at gzcompress and gzdeflateĮxample using decoded version of your text $original = "In other cases, some countries have gradually learned to produce the same products and services that previously only the U.S. Please Note neither base64 nor encryption was designed for reduction of string length. Then, I want to call encrypted data from MySQL in order to decrypt.Īny way to reduce the size of texts? Base64 does not work for me, is there any other way? I want to encrypt text data before storing. So, here is an example what I want to do: I'm thinking of encrypting and decrypting the data. First of all, I don't know how I can do this job. I want to reduce text's size before sending them to MySQL. QgdG8gcHJvZHVjZSB0aGUgc2FtZSBwcm9kdWN0cyBhbmQgc2VydmljZXMgdGhhdCBwcmV2ĪW91c2x5IG9ubHkgdGhlIFUuUy4gYW5kIGEgZmV3IG90aGVyIGNvdW50cmllcyBjb3VsZCīwcm9kdWNlLiBSZWFsIGluY29tZSBncm93dGggaW4gdGhlIFUuUy4gaGFzIHNsb3dlZC4= SW4gb3RoZXIgY2FzZXMsIHNvbWUgY291bnRyaWVzIGhhdmUgZ3JhZHVhbGx5IGxlYXJuZW However, I need to make searching and I can't search any data while they are located in a zip file. ![]() Actually it has worked, reduced the total size. There are about 50 billion different texts. In other cases, some countries have gradually learned to produce the same products and services that previously only the U.S. Description: I have a huge MySQL database table.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |