132 lines
No EOL
18 KiB
XML
132 lines
No EOL
18 KiB
XML
<?xml version="1.0" encoding="utf-8"?>
|
|
<?xml-stylesheet type="text/xsl" href="../assets/xml/rss.xsl" media="all"?><rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Caught in the Net (Posts about python)</title><link>francescomecca.eu</link><description></description><atom:link href="francescomecca.eu/categories/python.xml" rel="self" type="application/rss+xml"></atom:link><language>en</language><copyright>Contents © 2020 <a href="mailto:francescomecca.eu">Francesco Mecca</a> </copyright><lastBuildDate>Wed, 29 Jan 2020 10:04:36 GMT</lastBuildDate><generator>Nikola (getnikola.com)</generator><docs>http://blogs.law.harvard.edu/tech/rss</docs><item><title>Interpolation using a genetic algorithm</title><link>francescomecca.eu/blog/2016/5/15/genetic-alg/</link><dc:creator>Francesco Mecca</dc:creator><description><div><p>This weekend I was in Milan to get a visa and I had the opportunity to work with a friend, Michele, on genetic algorithms.
|
|
It was the first time I dig up in such field and it was very exciting.
|
|
In this post I want to explain some bits of our work.</p>
|
|
<h3>A brief introduction to GA</h3>
|
|
<p>A genetic algorithm is a search/optimization algorithm that uses an heuristic approach to reduce the search space and evolve gradually to a solution.</p>
|
|
<h5>Population</h5>
|
|
<p>It is an algorithm that has its root in the theory of natural selectioni by Charles Darwin.
|
|
The main components of a GA are:</p>
|
|
<ul>
|
|
<li>the population, that concentrate all the available solutions at a given time;</li>
|
|
<li>the fitness function, that gives an approximation of the quality of the solution codified by a given member of the population.</li>
|
|
</ul>
|
|
<p>In a GA the first thing to do is to generate a population.</p>
|
|
<p>A population is a group of objects with given attributes, usually a string, and they contains in some form the solution (usually inside a string); the first population is randomly generated and contains a big number of solutions, but not every solution (this is not a bruteforce approach).</p>
|
|
<p>After this step the fitness functions evaluates the quality of every solution that a given member carries: the evaluation should be considered from a bottom up point of view.</p>
|
|
<h5>Reproduction</h5>
|
|
<p>Now, as in Darwin's theory of evolution, the member of the population are going to "reproduce": two members are going to be coupled to generate a new member of the second generation and every child member will contain a solution that is the product of the original genes of their parent members.</p>
|
|
<p>This time the reproduction of the population into a second one is not entirely random. The fitness function gives us an approximation of the quality of every gene that a member carries and by the rule of the "survival by the fittest" the probability that a member is going to reproduce with another one is proportional to the quality of its genes.</p>
|
|
<p>When we have a second generation of members we can recur on our GA and generate a third generation. From this point we can recur until we converge to a solution that is common to every member, or at least that is suited to our needs.</p>
|
|
<h5>Mutation</h5>
|
|
<p>Actually, in some cases, a mutation function can be added, so that, like in real world, some times the genes are "scrambled" indipendently from the fitness function.</p>
|
|
<p>There is more to a GA, for example we could talk about possible ways of storing the genes inside a member or when to use mutation, anyway I want to stop here and continue with an analysis of my problem.</p>
|
|
<h3>Interpolating a function using a GA</h3>
|
|
<p>Me and Michele decided to spend some time developing a little python script to explore GA capabilities and we decided to interpolate some points on a cartesian plane.</p>
|
|
<p>Our program, that is available <a href="http://francescomecca.eu:3000/pesceWanda/interpol_genetica">here</a> uses a class to define the various members of the population and a string for the genes, a class as well for the points on the plane.</p>
|
|
<p>The fitness function is not as precise as it should be because this is only a proof of concept:</p>
|
|
<p>.. code:: python</p>
|
|
<pre class="code literal-block"><span></span>mutationProbability = 0.1
|
|
rangeLimit = 5
|
|
def fitness(item, pointList, n):
|
|
value = 0
|
|
for p in pointList:
|
|
y = 0
|
|
for i in range(n):
|
|
y += item.gene[i] * pow(p.x, i)
|
|
result = 1 - (abs (p.y - y) / rangeLimit)
|
|
if result &lt; 0:
|
|
result = 0
|
|
value += result
|
|
return value / n
|
|
</pre>
|
|
|
|
|
|
<p>item is just a member of the population, poinList is the list of points and n is the number of points (n - 1 is the grade of the function).</p>
|
|
<pre class="code literal-block"><span></span>for i in range(n):
|
|
y += item.gene[i] * pow(p.x, i)
|
|
</pre>
|
|
|
|
|
|
<p>this piece of code gives us the value of the function encoded in the genes in the points of pointList;</p>
|
|
<pre class="code literal-block"><span></span>result = 1 - (abs (p.y - y) / rangeLimit)
|
|
if result &lt; 0:
|
|
result = 0
|
|
</pre>
|
|
|
|
|
|
<p>while here the script stores 1 - the previous result because if the GA has yield a good result there should be distance = 0 from the function evaluated and the points; If this is the case, the fitness function should attribute the highest possible reproduction probability for that member.
|
|
At the end the fitness function returns the total value over the number of points evaluated.</p>
|
|
<p>As you can see this fitness function is by no means an optimal one. The reproduction probability is higher for functions that crosses some points and are really distant from others rather than for functions that are closer to every point but crosses none.
|
|
Anyway for simple cases the GA yields good results, as an example for points (0 0), (1 4), (2 9) one of the member with the highest reproduction probability has this function in its genes:</p>
|
|
<pre class="code literal-block"><span></span>-0.0487839869993989 * x^0 + 4.600339125358671 * x^1 + -0.2780958075230644 * x^2
|
|
</pre>
|
|
|
|
|
|
<p>that crosses this points: (0 -0.0488), (1 4.2735), (2 8.0395) given 80 iterations, initial population of 600 members and a two digit approximation.</p>
|
|
<p>For a more precise computation a higher population size and a really high number of iterations should be used.</p></div></description><category>AI</category><category>Genetic algorithm</category><category>PesceWanda</category><category>programming</category><category>python</category><guid>francescomecca.eu/blog/2016/5/15/genetic-alg/</guid><pubDate>Sun, 15 May 2016 00:00:00 GMT</pubDate></item><item><title>Kyuss Music Player</title><link>francescomecca.eu/blog/2016/4/17/kpd-player/</link><dc:creator>Francesco Mecca</dc:creator><description><div><p>For a long time I have been using Clementine music player on my workstation. Recently I reinstalled Gentoo on my desktop and I wanted to avoid installing QT libraries of any sort.
|
|
So I switched to <a href="https://www.musicpd.org/">mpd</a> and I have fallen in love with it. It is very flexible, fast and enriched by a lot of community software.
|
|
For some weeks I used mpc client as my primary client for mpd but I was not satisfied with it. Even though it is pretty minimal but packed with every feature mpd permits, the search feels uncomfortable because is case sensitive and need artist, album, etc. flags before any entry.
|
|
This is why I have written kpd together with Francesco Gallà</p>
|
|
<h3>Kyuss Player Client</h3>
|
|
<p>kpd is an acronym for Kyuss Player Client because we have been listening only to <a href="https://en.wikipedia.org/wiki/Kyuss">Kyuss</a> while programming this client.
|
|
We have reimplemented the search functions to suit our habits. No more case sensitive, optional 'artist, album, title' flags.
|
|
kpd accepts only one string as the search argument and implements optional filter arguments to narrow the search in a grep like way.
|
|
I welcome you to read the <a href="http://francescomecca.eu:3000/pesceWanda/kpd">readme</a> in my git to understand how the search works.
|
|
Anyway in this post I want to explain bits of the code.</p>
|
|
<h4>Main</h4>
|
|
<p>The main kpd file invoked when the command is run in the console is kpd.py
|
|
The most interesting part in this file IMHO is these lines:</p>
|
|
<p>.. code:: python</p>
|
|
<pre class="code literal-block"><span></span> for el in argsOrder:
|
|
if dictArgs[el] != False:
|
|
client.update_status ()
|
|
methodToCall = getattr (util, el)
|
|
retUtil = methodToCall (client, dictArgs[el], searchRes)
|
|
</pre>
|
|
|
|
|
|
<p>argsOrder is a list of the arguments on the command line in the order the user wrote them.
|
|
kpd uses a dictionary to store for every argument the corrispective string for the function that will be invoked using getattr.
|
|
In this way any argument can be added to the main file without writing any other line of code. WE used this method to avoid using switch alike solutions.</p>
|
|
<h4>Util</h4>
|
|
<p>The util.py source file is a pretty easy source file to read. It contains every function that can be invoked by command line arguments. Every function has the same 'prototypes' so that they can be called using the method explained above.
|
|
To implement <code>no-output</code> and <code>output</code> function I have used a class:
|
|
to suppress the output on the console the program assign to <em>sys.stdout</em> a dummy class that save the original stdout on a variable and replaces write and flush functions so that they are just pass. and no output is written.
|
|
To permit output after suppression the program just reassing the original value to sys.stdout.</p>
|
|
<h4>Database Search</h4>
|
|
<p>In MPDdatabase.py we have written the search functions.
|
|
Originally we intended to just read and import in a dictionary the whole mpd database that is stored compressed in the home directory.
|
|
This list of dictionaries stores every entry related to the song and if any of them matches the search string or the filter string (considering also flags if any) the related song is printed on the output and saved in a list so it can be added by the add function.
|
|
This approach result very efficent in term of precision but it lacked speed. For a database of about 77 thousand songs (about 550k lines) a search query could last almost 2 seconds.
|
|
To improve the speed of the search we used the pickle module. The pickle module allows kpd to dump the data structure used to store the database in memory on a file that can be read easily by using the <code>pickle.load</code> function.
|
|
In this way the search lasts about 40 milliseconds on the same database that wastes about 16MiB of memory on disk.</p>
|
|
<h3>Conclusion</h3>
|
|
<p>This was really fun. It was our first hand on python project and the first real program we have written since we started learning programming at our university.
|
|
I discovered that programming allows me to relax and that is really cool to have custom software for activities you do every day.
|
|
The source for our program is stored in my git <a href="http://francescomecca.eu:3000/pesceWanda/kpd">here</a> and you are free to modify it.</p></div></description><category>mpd</category><category>music player</category><category>PesceWanda</category><category>programming</category><category>python</category><guid>francescomecca.eu/blog/2016/4/17/kpd-player/</guid><pubDate>Sun, 17 Apr 2016 00:00:00 GMT</pubDate></item><item><title>The Buridan's donkey in python</title><link>francescomecca.eu/blog/2016/4/2/buridan_donkey/</link><dc:creator>Francesco Mecca</dc:creator><description><div><p>During the final weeks of my exam session I started reading a bit about python 3 using an excellent book: <a href="http://www.diveintopython.net/">Dive into Python</a>.
|
|
When I noted that python uses the <a href="https://en.wikipedia.org/wiki/Mersenne_Twister">Mersenne Twister PRNG</a> as well I decided to write another version of my <a href="http://francescomecca.eu/index.php/archives/207">Buridan's donkey program</a>.</p>
|
|
<p>.. code:: python</p>
|
|
<pre class="code literal-block"><span></span> <span class="s s-Atom">import</span> <span class="s s-Atom">random</span><span class="p">,</span> <span class="s s-Atom">sys</span>
|
|
|
|
<span class="s s-Atom">if</span> <span class="k">__</span><span class="s s-Atom">name__</span> <span class="o">==</span> <span class="s s-Atom">'__main__':</span>
|
|
<span class="s s-Atom">args</span> <span class="o">=</span> <span class="nf">list</span><span class="p">()</span>
|
|
<span class="s s-Atom">if</span> <span class="o">not</span> <span class="s s-Atom">sys</span><span class="p">.</span><span class="s s-Atom">stdin</span><span class="p">.</span><span class="nf">isatty</span><span class="p">()</span><span class="s s-Atom">:</span>
|
|
<span class="s s-Atom">for</span> <span class="s s-Atom">line</span> <span class="s s-Atom">in</span> <span class="s s-Atom">sys</span><span class="p">.</span><span class="nn">stdin</span><span class="p">:</span>
|
|
<span class="s s-Atom">if</span> <span class="s s-Atom">line</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">is</span> <span class="s s-Atom">'\n':</span>
|
|
<span class="s s-Atom">line</span> <span class="o">=</span> <span class="s s-Atom">line</span><span class="p">[:-</span><span class="mi">1</span><span class="p">]</span>
|
|
<span class="s s-Atom">args</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="s s-Atom">line</span><span class="p">)</span>
|
|
<span class="nn">else</span><span class="p">:</span>
|
|
<span class="s s-Atom">args</span> <span class="o">=</span> <span class="s s-Atom">sys</span><span class="p">.</span><span class="s s-Atom">argv</span><span class="p">[</span><span class="mi">1</span><span class="s s-Atom">:</span><span class="p">]</span>
|
|
<span class="s s-Atom">argRange</span> <span class="o">=</span> <span class="nf">len</span><span class="p">(</span><span class="s s-Atom">args</span><span class="p">)</span>
|
|
<span class="s s-Atom">for</span> <span class="s s-Atom">i</span> <span class="s s-Atom">in</span> <span class="nf">range</span><span class="p">(</span><span class="s s-Atom">argRange</span><span class="p">)</span><span class="s s-Atom">:</span>
|
|
<span class="nf">print</span><span class="p">(</span><span class="nf">str</span><span class="p">(</span><span class="s s-Atom">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="o">+</span> <span class="s s-Atom">'.'</span><span class="p">,</span> <span class="s s-Atom">args</span><span class="p">.</span><span class="nf">pop</span><span class="p">(</span><span class="s s-Atom">random</span><span class="p">.</span><span class="nf">randrange</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nf">len</span><span class="p">(</span><span class="s s-Atom">args</span><span class="p">))))</span>
|
|
</pre>
|
|
|
|
|
|
<p>This script works in a different way than the one in c++.
|
|
Rather than shuffling a list made by the entries in the arguments, it pops randomly one entry from the list till the list is empty.</p>
|
|
<p>Not satisfied enough, I wrote also a telegram bot using the <a href="https://github.com/eternnoir/pyTelegramBotAPI">telebot library</a> that works as the script above but inside the telegram app.
|
|
The bot can be added to your contact list by simply searching for <a href="http://telegram.me/duridan_donkey_bot">@duridan_donkey_bot</a> (yes, a typo!)</p>
|
|
<p>All the code is opensource and can be found on my github page.</p>
|
|
<p>Francesco Mecca</p></div></description><category>buridan donkey</category><category>mersenne twister</category><category>PesceWanda</category><category>python</category><category>random</category><guid>francescomecca.eu/blog/2016/4/2/buridan_donkey/</guid><pubDate>Sat, 02 Apr 2016 00:00:00 GMT</pubDate></item></channel></rss> |