Update: I am (as of August 2018) once again gainfully employed.

Since a slightly-wider circle of people know this now, it’s time to just go public with the news: my last day at now-former employer was toward the end of January. At the time I told a few friends, but wasn’t in a huge rush to find something else immediately; I’d been getting ready to depart for a little while, and there were some things ...

Read full entry

It would be ideal to have a backward compatibility between different versions of programs and programming languages. However, it’s not the case and we have to consider this when supporting old products. In this article, you’ll learn how Python 2 differs from Python 3, and how to translate old programs written in the second version of the language to Python 3.

I haven’t spoken on our PyDataLondon meetup community in a while so I figure a few numbers are due. We’re now at an incredible 7,800 members and just this month we had 200 members in the room at AHL’s new venue. We’re a volunteer run community – you’ll see the list of our brilliant volunteers here along with their Twitter accounts.

I polled the attendees this month and 1/3 of the hands went up (see below) to the question “Who is a first-timer to this meetup?”. This shows the continued growth in our community and in the wider data science ecosystem in London. Welcome along!

One of our talks was on Pandas v1, that included an update by Marc on how Python 2.x is being deprecated in Pandas next year and the new Cyberpandas and Fletcher (dtype extensions including faster strings) libraries. Marc also noted that Pandas is estimated to have 5-10 million users! One of the benefits of internal Pandas updates will be the “UInt8” and related dtypes – we’ll have integers with NaNs for the first time ever (previously int arrays with NaNs were promoted to floats which have NaN support in numpy).

Given the continued growth of our ecosystem – this means we have more Python newbies and more Data Science newbies (including converts moving away from Excel and SPSS). We’re always looking for new speakers. New speakers don’t have to be experienced data scientists – a 5 minute lightning talk on how you are transitioning in to this ecosystem from elsewhere can be hugely valuable to other new members. A talk (5 mins or 30 minutes) on a technique you’re experienced in – even if there’s no equivalent Python library – is also incredibly educational. Please come and share your knowledge.

Talking will raise your profile and it’ll raise your employer’s profile (if that’s what you’re after) and that obviously helps with hiring. We continue after the meetup in the local pub (typically The Banker) so anyone who’s been speaking and who ends with “and we’re hiring!” tends to have interesting conversations in the pub afterwards. With 200 attendees it isn’t hard to find folk who’d be interested in your role. Remember – this is most effective for speakers as you have the entire audience’s attention. You’ll find instructions here on how to submit a talk.

AHL continue to support our open source PyData world (along with other open events like the London Machine Learning meetup), they now rent a professional auditorium next to their building each month for us with full hosting, mics on every chair and video recording (see them at PyDataTV) for speakers who consent. This isn’t cheap of course and it provides evidence of the growth of Python’s Data Science stack in the London financial community. AHL’s activity at the meetup is to say a few words before the break about who they’re hiring for, before everyone heads out for more beer. Thanks AHL for your continued support! You might also want to check their github repo.

There’s a whole pile of PyData conferences coming up, you might find some are closer to you or your offices than you imagine. Go check the list. The PyData meetup ecosystem has grown world-wide to 111 events now too!

PyData of course is supported by NumFOCUS, the non-profit in the US. NumFOCUS backs a lot of our open source tools. They’re having a summit late this September in the US – everyone is welcome, if you’re interested in the deeper direction of Python and the Data Science community then you might want to attend (or send a representative from your group?).

Of course you might also want to be hired by a company that works in our PyData ecosystem. I post out jobs (UK-centric but they stretch to western Europe and sometimes to the US) every 2 weeks to 650+ data scientists and engineers, typically 7 roles (mostly permie, some contract, all Python focused). You might want to join that list (note your email is always kept private and is never shared). Attending PyData members (i.e. anyone who helps build our ecosystem) gets a first post gratis.

The post On the growth of our PyDataLondon community appeared first on Entrepreneurial Geekiness.

Google services are cool and you can build products and services around it. We will see through examples how you can use various google services such as spreadsheet, slides and drive through Python. I hope people can take ideas from the following example to do amazing stuffs with Google services. There is a part one […]

The post Google APIs and Python – Part II appeared first on The Tara Nights.

Python is a wonderful language for both beginners and expert programmers, but getting started can be daunting. Which version should I use? Which editors are best? What do you mean there are different implementations and environments? Here's a guide to help navigate these big FAQs.

tl;dr

For most people:

• Use the latest version of Python 3.
• Use the CPython implementation.
• Use pipenv to manage packages and installations.
• Use Visual Studio Code or PyCharm for editing code.

Which Version?

Python 2 and Python 3 are actually different languages. The differences go deeper than just print statements. The What's New in Python page on the official doc site lists all the gory details, and decent articles showcasing differences can be found here, here, and here. Although Python 3 is newer, Python 2 remains prevalent. Most popular packages use Python packaging tools to support both versions. The Python Wiki makes it clear that Python 3 is the better choice:

Python 2.x is legacy, Python 3.x is the present and future of the language

Furthermore, Python 2 will reach end-of-life in 2020. The Python team will continue to provide bug fixes for 2.7 until 2020 (PEP 373), but there will be no new language features and no 2.8 (PEP 404). (Originally, end-of-life was planned for 2015, but it was pushed back by 5 years.) There is even a Python 2.7 Countdown clock online.

Which Implementation?

In purest terms, "Python" is a language specification. An implementation provides the language processing tools (compiler, interpreter, etc.) to run Python programs. The Hitchhiker's Guide to Python has a great article entitled Picking an Interpreter that provides a good summary of available interpreters. Others are listed on python.org and the Python Wiki. The table below provides a quick overview of the big ones.

CPython

• most widely used implementation
• the reference implementation
• has the most libraries and support
• implemented in C
• supports Python 2 and 3

PyPy

• much faster than CPython
• much more memory efficient
• implemented in RPython
• supports Python 2 and 3

Jython

• implemented in Java
• runs on the JVM
• supports Python 2
• only a sandbox for Python 3
• no project updates since May 2015

IronPython

• implemented for .NET
• lets Python libs call .NET and vice versa
• supports Python 2

Python for .NET

• integrates CPython with .NET/Mono runtime
• supports Python 2 and 3

Stackless Python

• branch of CPython with real threading

MicroPython

• optimized for microcontrollers
• uses a subset of the standard library

Unless you have a very specific reason, just use CPython. In fact, most people are referring to CPython when they say "Python." CPython has the most compatibility, the widest package library, and the richest support. If you really need speed, consider PyPy.

Managing Installations

pip is the standard tool for installing Python packages. The simplest way to install Python is to install it "globally" for the system. In fact, some operating systems like macOS and Ubuntu have Python pre-installed. However, global installation has limitations:

1. You may want to develop packages for both versions 2 and 3.
2. You may not have permissions to add new packages globally.
3. Different projects may require different versions of packages.

These problems can be solved by using "virtual" environments. A virtual environment is like a local Python installation with a specific package set. For example, I have created virtual environments for Python as part of Jenkins build jobs, since I did not have permission to install special automation packages globally on the Jenkins slaves.

The standard virtual environment tool for Python is venv, which has been packaged with (C)Python since 3.3. (venv had a command line wrapper named pyvenv, but this was deprecated in 3.6.) Another older but still popular third-party tool is virtualenv. As explained in this Reddit post, venv _is the Python-sanctioned replacement for _virtualenv. However, virtualenv supports Python 2, whereas venv does not. Conda is an environment manager popular with the science and data communities, and it can support other languages in addition to Python.

That being said, there is a relatively new package manager taking the Python world by storm: pipenv. In short time, it became the officially-recommended tool from Python.org. Pipenv combines pip, Pipfile, and virtualenv into an easy workflow with simple commands. My recommendation is to use pipenv for all new projects.

Editors and IDEs

After setting up your Python environment, you are ready to start programming! There are two routes to take for text editing: source code editors and integrated development environments.

Source code editors are lightweight but often include basics like syntax highlighting and basic auto-completion. They're great for quick edits and light scripting. Many have plugins. Popular choices are Visual Studio Code, Sublime, Atom, and Notepad++. My current favorite is Visual Studio Code because the Python extensions are stellar and settings are simple - just remember to install the extensions you need! I use it personally for Django development.

For more intense development, I highly recommend an IDE like JetBrains PyCharm, PyDev for Eclipse, Wing Python IDE, or Eric. IDEs provide rich development support, especially for larger apps that use frameworks like Django, Pyramid, and SQLAlchemy. They also make testing easier with plugins for test frameworks like pytest, behave, and others. PyCharm and PyDev are particularly nice because they can integrate into their larger IDEs (IntelliJ IDEA and Eclipse, respectively) to handle more languages. Personally, I prefer PyCharm, but advanced features require a paid license.

Pythonese

The Python community throws around a few terms you should know:

Pythonic

• describes idiomatic code for Python
• closely related to conciseness, readability, and elegance
• highly recommended to use
• follow style guidelines

Pythonista

• someone who loves the Python language
• often an advanced Python programmer

Pythoneer

• a programmer who uses Python to solve problems

The Zen of Python

• the list of guiding principles for Python's design
• run "import this" to see them

The Python Software Foundation (PSF)

• non-profit org
• keeps Python going strong
• support them!

PyCon

• the annual Python conference held in North America
• GO - it will change your life!
• several other conferences are held worldwide

Benevolent Dictator for Life(BDFL)

• Guido van Rossum
• the inventor of Python
• resigned in July 2018 but remains BDFL Emeritus

This article was originally posted at AutomationPanda.com and has been reposted here with permission.

Keep Calm and Code in Python!

Andy

This blog has been dormant for too long – it’s time to post another article. In order to live up to the blog title, this article will combine some old and seasoned stuff (PostgreSQL and Shakespeare’s works) with the latest trends for agile developers (GraphQL, asyncio and type hints for Python).

Introduction

Text preprocessing is one of the most important tasks in Natural Language Processing (NLP). For instance, you may want to remove all punctuation marks from text documents before they can be used for text classification. Similarly, you may want to extract numbers from a text string. Writing manual scripts for such preprocessing tasks requires a lot of effort and is prone to errors. Keeping in view the importance of these preprocessing tasks, the Regular Expressions (aka Regex) have been developed in different languages in order to ease these text preprocessing tasks.

A Regular Expression is a text string that describes a search pattern which can be used to match or replace patterns inside a string with a minimal amount of code. In this tutorial, we will implement different types of regular expressions in the Python language.

To implement regular expressions, the Python's re package can be used. Import the Python's re package with the following command:

import re  

Searching Patterns in a String

One of the most common NLP tasks is to search if a string contains a certain pattern or not. For instance, you may want to perform an operation on the string based on the condition that the string contains a number.

To search a pattern within a string, the match and findall function of the re package is used.

The match Function

Initialize a variable text with a text string as follows:

text = "The film Titanic was released in 1998"

Let's write a regex expression that matches a string of any length and any character:

result = re.match(r".*", text)  

The first parameter of the match function is the regex expression that you want to search. Regex expression starts with the alphabet r followed by the pattern that you want to search. The pattern should be enclosed in single or double quotes like any other string.

The above regex expression will match the text string, since we are trying to match a string of any length and any character. If a match is found, the match function returns _sre.SRE_Match object as shown below:

type(result)  

Output:

_sre.SRE_Match  

Now to find the matched string, you can use the following command:

result.group(0)  

Output:

'The film Titanic was released in 1998'  

In case if no match is found by the match function, a null object is returned.

Now the previous regex expression matches a string with any length and any character. It will also match an empty string of length zero. To test this, update the value of text variable with an empty string:

text = ""

Now, if you again execute the following regex expression, a match will be found:

result = re.match(r".*", text)  

Since we specified to match the string with any length and any character, even an empty string is being matched.

To match a string with a length of at least 1, the following regex expression is used:

result = re.match(r".+", text)  

Here the plus sign specifies that the string should have at least one character.

Searching Alphabets

The match function can be used to find any alphabet letters within a string. Let's initialize the text variable with the following text:

text = "The film Titanic was released in 1998"

Now to find all the alphabet letter, both uppercase and lowercase, we can use the following regex expression:

result = re.match(r"[a-zA-z]+", text)  

This regex expression states that match the text string for any alphabets from small a to small z or capital A to capital Z. The plus sign specifies that string should have at least one character. Let's print the match found by the above expression:

print(result.group(0))  

Output:

The  

In the output, you can see that the first word i.e. The is returned. This is because the match function only returns the first match found. In the regex we specified that find the patterns with both small and capital alphabets from a to z. The first match found was The. After the word The there is a space, which is not treated as an alphabet letter, therefore the matching stopped and the expression returned just The, which is the first match.

However, there is a problem with this. If a string starts with a number instead of an alphabet, the match function will return null even if there are alphabets after the number. Let's see this in action:

text = "1998 was the year when the film titanic was released"  
result = re.match(r"[a-zA-z]+", text)  
type(result)  

Output:

NoneType  

In the above script, we have updated the text variable and now it starts with a digit. We then used the match function to search for alphabets in the string. Though the text string contains alphabets, null will be returned since match function only matches the first element in the string.

To solve this problem we can use the search function.

The search Function

The search function is similar to the match function i.e. it tries to match the specified pattern. However, unlike the match function, it matches the pattern globally instead of matching only the first element. Therefore, the search function will return a match even if the string doesn't contain an alphabet at the start of the string but contains an alphabet elsewhere in the string, as shown below:

text = "1998 was the year when the film titanic was released"  
result = re.search(r"[a-zA-z]+", text)  
print(result.group(0))  

Output:

was  

The search function returns "was" since this is the first match that is found in the text string.

Matching String from the Start

To check if a string starts with a specific word, you can use the carrot key i.e. ^ followed by the word to match with the search function as shown below. Suppose we have the following string:

text = "XYZ 1998 was the year when the film titanic was released"

If we want to find out whether the string starts with "1998", we can use the search function as follows:

result = re.search(r"^1998", text)  
type(result)  

In the output, null will be returned since the text string doesn't contain "1998" directly at the start.

Now let's change the content text variable and add "1998" at the beginning and then check if "1998" is found at the beginning or not. Execute the following script:

text = "1998 was the year when the film titanic was released"  
if re.search(r"^1998", text):  
    print("Match found")
else:  
    print("Match not found")

Output:

Match found  

Matching Strings from the End

To check whether a string ends with a specific word or not, we can use the word in the regular expression, followed by the dollar sign. The dollar sign marks the end of the statement. Take a look at the following example:

text = "1998 was the year when the film titanic was released"  
if re.search(r"1998$", text):  
    print("Match found")
else:  
    print("Match not found")

In the above script, we tried to find if the text string ends with "1998", which is not the case.

Output:

Match not found  

Now if we update the string and add "1998" at the end of the text string, the above script will return ‘Match found' as shown below:

text = "was the year when the film titanic was released 1998"  
if re.search(r"1998$", text):  
    print("Match found")
else:  
    print("Match not found")

Output:

Match found  

Substituting text in a String

Till now we have been using regex to find if a pattern exists in a string. Let's move forward with another advanced regex function i.e. substituting text in a string. The sub function is used for this purpose.

Let's take a simple example of the substitute function. Suppose we have the following string:

text = "The film Pulp Fiction was released in year 1994"

To replace the string "Pulp Fiction" with "Forrest Gump" (another movie released in 1994) we can use the sub function as follows:

result = re.sub(r"Pulp Fiction", "Forrest Gump", text)  

The first parameter to the sub function is the regular expression that finds the pattern to substitute. The second parameter is the new text that you want as a replacement for the old text and the third parameter is the text string on which the substitute operation will be performed.

If you print the result variable, you will see the new string.

Now let's substitute all the alphabets in our string with character "X". Execute the following script:

text = "The film Pulp Fiction was released in year 1994"  
result = re.sub(r"[a-z]", "X", text)  
print(result)  

Output:

TXX XXXX PXXX FXXXXXX XXX XXXXXXXX XX XXXX 1994  

It can be seen from the output that all the characters have been replaced except the capital ones. This is because we specified a-z only and not A-Z. There are two ways to solve this problem. You can either specify A-Z in the regular expression along with a-z as follows:

result = re.sub(r"[a-zA-Z]", "X", text)  

Or you can pass the additional parameter flags to the sub function and set its value to re.I which refers to case insensitive, as follows:

result = re.sub(r"[a-z]", "X", text, flags=re.I)  

More details about different types of flags can be found at Python regex official documentation page.

Shorthand Character Classes

There are different types of shorthand character classes that can be used to perform a variety of different string manipulation functions without having to write complex logic. In this section we will discuss some of them:

Removing Digits from a String

The regex expression to find digits in a string is \d. This pattern can be used to remove digits from a string by replacing them with an empty string of length zero as shown below:

text = "The film Pulp Fiction was released in year 1994"  
result = re.sub(r"\d", "", text)  
print(result)  

Output:

The film Pulp Fiction was released in year  

Removing Alphabet Letters from a String

text = "The film Pulp Fiction was released in year 1994"  
result = re.sub(r"[a-z]", "", text, flags=re.I)  
print(result)  

Output:

1994  

Removing Word Characters

If you want to remove all the word characters (letters and numbers) from a string and keep the remaining characters, you can use the \w pattern in your regex and replace it with an empty string of length zero, as shown below:

text = "The film, '@Pulp Fiction' was ? released in % $ year 1994."  
result = re.sub(r"\w","", text, flags = re.I)  
print(result)  

Output:

, '@ '  ?   % $  .

The output shows that all the numbers and alphabets have been removed.

Removing Non-Word Characters

To remove all the non-word characters, the \W pattern can be used as follows:

text = "The film, '@Pulp Fiction' was ? released in % $ year 1994."  
result = re.sub(r"\W", "", text, flags=re.I)  
print(result)  

Output:

ThefilmPulpFictionwasreleasedinyear1994  

From the output, you can see that everything has been removed (even spaces), except the numbers and alphabets.

Grouping Multiple Patterns

You can group multiple patterns to match or substitute in a string using the square bracket. In fact, we did this when we matched capital and small letters. Let's group multiple punctuation marks and remove them from a string:

text = "The film, '@Pulp Fiction' was ? released _ in % $ year 1994."  
result = re.sub(r"[,@\'?\.$%_]", "", text, flags=re.I)  
print(result)  

Output:

The film Pulp Fiction was  released  in   year 1994  

You can see that the string in the text variable had multiple punctuation marks, we grouped all these punctuations in the regex expression using square brackets. It is important to mention that with a dot and a single quote we have to use the escape sequence i.e. backward slash. This is because by default the dot operator is used for any character and the single quote is used to denote a string.

Removing Multiple Spaces

Sometimes, multiple spaces appear between words as a result of removing words or punctuation. For instance, in the output of the last example, there are multiple spaces between in and year. These spaces can be removed using the \s pattern, which refers to a single space.

text = "The film      Pulp Fiction      was released in   year 1994."  
result = re.sub(r"\s+","", text, flags = re.I)  
print(result)  

Output:

The film Pulp Fiction was released in year 1994.  

In the script above we used the expression \s+ which refers to single or multiple spaces.

Removing Spaces from Start and End

Sometimes we have a sentence that starts or ends with a space, which is often not desirable. The following script removes spaces from the beginning of a sentence:

text = "         The film Pulp Fiction was released in year 1994"  
result = re.sub(r"^\s+", "", text)  
print(result)  

Output:

The film Pulp Fiction was released in year 1994  

Similarly, to remove space at the end of the string, the following script can be used:

text = "The film Pulp Fiction was released in year 1994      "  
result = re.sub(r"\s+$", "", text)  
print(result)  

Removing a Single Character

Sometimes removing punctuation marks, such as an apostrophe, results in a single character which has no meaning. For instance, if you remove the apostrophe from the word Jacob's and replace it with space, the resultant string is Jacob s. Here the s makes no sense. Such single characters can be removed using regex as shown below:

text = "The film Pulp Fiction     s was b released in year 1994"  
result = re.sub(r"\s+[a-zA-Z]\s+", "", text)  
print(result)  

Output:

The film Pulp Fiction was released in year 1994  

The script replaces any small or capital letter between one or more spaces, with a single space.

Splitting a String

String splitting is another very important function. Strings can be split using split function from the re package. The split function returns a list of split tokens. Let's split a string of words where one or more space characters are found, as shown below:

text = "The film      Pulp   Fiction was released in year 1994      "  
result = re.split(r"\s+", text)  
print(result)  

Output:

['The', 'film', 'Pulp', 'Fiction', 'was', 'released', 'in', 'year', '1994', '']

Similarly, you can use other regex expressions to split a string using the split functions. For instance, the following split function splits string of words when a comma is found:

text = "The film, Pulp Fiction, was released in year 1994"  
result = re.split(r"\,", text)  
print(result)  

Output:

['The film', ' Pulp Fiction', ' was released in year 1994']

Finding All Instances

The match function conducts a match on the first element while the search function conducts a global search on the string and returns the first matched instance.

For instance, if we have the following string:

text = "I want to buy a mobile between 200 and 400 euros"

We want to search all the digits from this string. If we use the search function, only the first occurrence of digits i.e. 200 will be returned as shown below:

result = re.search(r"\d+", text)  
print(result.group(0))  

Output:

200  

On the other hand, the findall function returns a list that contains all the matched utterances as shown below:

text = "I want to buy a mobile between 200 and 400 euros"  
result = re.findall(r"\d+", text)  
print(result)  

Output:

['200', '400']

You can see from the output that both "200" and "400" is returned by the findall function.

Conclusion

In this article we studied some of the most commonly used regex functions in Python. Regular expressions are extremely useful for preprocessing text that can be further used for a variety of applications, such as topic modeling, text classification, sentimental analysis, and text summarization, etc.

1. Problems with using a rough greyscale algorithm? - [37/8]
2. Collatz Conjecture Python - Incorrect Output Above 2 Trillion (Only!) - [28/1]
3. if-else vs "or" operation for None-check - [15/6]
4. Faster Python technique to count triples from a list of numbers that are multiples of each other - [14/5]
5. Python: What happens if script stops while requests.get() is executing? - [13/1]
6. Extending a class by parameter in Python - [12/2]
7. How do I shorten this boolean expression? - [10/6]
8. Getting outer environment arguments from java using graal python - [8/1]
9. Is the File-Object iterator "broken?" - [8/1]
10. Swapping list elements in python where the expressions contain function calls - [8/1]

In the previous part of this series, we started to get familiar with telepot, a Python module to interact with Telegram bots and had a short look at how the Telegram bot API is providing messages as JSON structures.

Today we will talk about the Webhook-method (push) instead of the previously introduced getUpdate-method (pull).

What’s the difference in getUpdates and a webhook? 

The Default: getUpdates (pull) 

By default (when a new bot is registered), a bot is configured to cache messages until they are actively fetched by some request (getUpdates). This is also called “pull method”.
When a message is sent to the bot while it is configured for this mode, it is cached in some kind of buffer within the infrastructure of Telegram (until they are delivered to the bot, but not longer than 24 hours, as described in the docs).

While this mode offers an easy start because you do not need to have any infrastructure or code prepared to create the bot and start exchanging messages with it, it has some downsides:

• You have to take care of having your code to ask for updates frequently.
• You will often ask for updates while there is none, generating unnecessary load and traffic.
• You do not have the opportunity to implement a real-time service since updates will always be delayed until the next update execution.

There might be a good reason to keep it like this nevertheless, like if you do not have a way to meet the technical requirements for the alternative method which is a webhook, like that you do not have a way to expose an interface to the public internet for example or if you want to prepare and test some code before firing up a deployed service. But most serious bots probably will change this to the alternate webhook (push) method sooner or later.

GET and POST HTTP methods

Before we look at the webhook push method, let’s first refresh some basics about how HTTP works. This will only cover some basics to better understand what is going on in the next sections. There is more to say and explain about the methods shown here and there are more methods to the protocol as well; not because of no reason there are 656 pages to O’Reilly’s book “HTTP: The Definitive Guide“. Since obviously, this is beyond the scope of this article, if you want to know more, please help yourself.

GET

When you are surfing the web with your favorite browser, GET requests are sent by that browser to the web servers providing your favorite websites, asking for a specific resource (like “

GET / HTTP/1.1

"GET /create-your-own-telegram-bot-with-django-on-heroku-part-3/?somethingIWantToSet=HubbaHubba HTTP/1.1"

followed by a myriad of additional requests for each asset (like CSS files, JS, images, …). The web server then does it’s magic and delivers the associated content in its answer; classically: A webpage.

The “?” in that request string separates the URL from an optional, additional section in that GET request: The query-string (

?somethingIWantToSet=HubbaHubba

http://www.my-great-site.com/index.php?theme=dark

To sum this up: GET is used to request data from a specified resource.

POST

Nowadays, most websites are not delivering just static content anymore. Instead, you can interact with them. You can, for example, send an email by filling a contact form or publish an answer to some forum’s thread or similar. To make this work, you need to send data to the remote website instead of sending a request to the remote web server to have something sent to you. The method in the HTTP protocol enabling you to do so is called POST.

If, for example, somebody is sending me an email using the contact form on my website, I can find a request like this in the access logs of my server:

162.158.89.81 - - [18/Aug/2018:13:11:30 +0200] "POST /contact/ HTTP/1.1" 200 21185 "https://www.marc-richter.info/contact/""Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0"

There is some detail in that, like the HTTP status (

200

https://www.marc-richter.info/contact/

Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0

POST /contact/ HTTP/1.1

This looks pretty much the same as the GET request is, isn’t it?

… wait a moment: Has somebody sent you an empty mail?

No. In fact, I sent it to myself with the following content:

To: Marc Richter
Name: Marc Richter
Email: noreply@marc-richter.info
Subject: Testmail

Message:
Hi Marc,
great site!
BR

The reason why you do not see this in the logline before is not that I cut that part. It doesn’t show up there, because unlike with GET requests, the POST request is just initiating the communication. The content/payload of the request is not part of the initializing request itself but is appended to the header of the request.

This is not only more secure to exchange data with the server, but also it is more convenient to exchange bigger amounts of data this way. Imagine, that all data which is ever sent to your favorite social network can be read by any admin who can read the log files of that server … or your employer since you are routed over hit proxy servers … 

To understand webhooks and to sum this up: Telegram bots are sending the JSON data we already saw in the previous part of this series to any URL we configure to them, using POST requests. The JSON data is then part of these request’s payload and can easily be extracted by Django (or any other application).

Again: If you want to get additional details on this, look them up for yourself; to understand how webhooks are working, what I explained so far is sufficient.

Webhook (push)

Now that we know what POST is and how it is working, let’s talk about how webhooks work.
There is a section in the official bot guide of Telegram, which explains how Webhooks work in three different levels of details; if you want to understand more details about Webhooks, I recommend reading at least the shortest version of it; I think it really is a great start for anybody not familiar with the concept of a webhook.

In my own words: A webhook is primarily a normal HTTP(S) socket, provided by a web server – like software listening to it for HTTP requests. Some piece of software is picking up everything which is sent towards this interface and applies some pre-defined logic on the data received. What logic applies is defined by your code.
This happens in real-time and is aware of the delivery result: If the message could not be forwarded to the defined interface, maybe because the server is down or there are temporary connection issues, Telegram cached the messages just like with the getUpdate pull method and tries is again after some time. As soon as the message was handed over to your interface successfully, Telegram forgets about the message and will never re-transmit it to your bot.

Giving webhooks a try

Enough of the boring theory: I bet you are desperate to see this whole stuff in action, don’t you? Let’s change your bot to webhook-mode and have everything sent to it forwarded to examine what we need to prepare our bot application for. After that, we will switch back to the getUpdates pull method again.

Attention: Please be aware that everything that will be sent to the bot will be forwarded to a 3rd party’s web service, I do have no control of. Please make sure you are OK with any data privacy policy of your country and that service before following the next lines, since I do not take responsibility about these things (if you revealed your online banking TAN lists because you felt like your bot could be interested in them or your partner is sending nudes unaware of this change, for example ).

First, you need to visit https://webhook.site in your favorite browser. You will be presented a page telling you all the details to utilize a unique webhook which just got created for you automatically without any registration – pretty awesome, isn’t it?

Copy that link and click the “Open in a new tab“-link once. That will open a new tab, showing an empty page; but – wait! The welcome page of your webhook will have changed immediately, listing a new GET request to have just arrived. That was your browser opening that empty page.
It works the same with POST or any other HTTP method anybody makes towards that interface.

Let’s now configure our bot’s webhook to use that URL you just copied as an destination interface for it’s webhook. You will need two things:

1. BOT_TOKEN: The URL of the webhook you just created
2. WEBHOOK_URL: The Token of your bot, we created in the previous sections of this series and I advised you to take a note on it because we will need that later; now is “later” 

In your browser, you now navigate to the following URL after you have crafted the full URL:

https://api.telegram.org/bot{BOT_TOKEN}/setWebhook?url={WEBHOOK_URL}

You need to replace the two “

{}

{"ok":true,"result":true,"description":"Webhook was set"}

And: You are good to go!
Make sure the page of your webhook is opened in your browser and then, send something to your bot in Telegram. Immediately (!) you should see that new message as a POST request in the “requests” bar on the left of your webhook page. You can tick the “Format JSON” checkbos at the upper right corner of that page to make the JSON data a bit easier to read.
And that’s it! This is a preview of how the messages which are sent to your bot will be forwarded to your webhook.

Go on, play around with that. Have different datatypes (like images, text, vcards) sent to your bot and check out how it will arrive at the webhook.

You can rely on this format will be the same which are about to hit your Python code, soon. Better be prepared for everything 

Switching back to getUpdate pull method

When you are done, better switch back to the getUpdate pull method, to not accidentally expose sensitive data to this publically available service.

This can be done, sending another GET request to the bot’s API like we used to enable the hook. The only difference is that we won’t provide a webhook URL this time:

https://api.telegram.org/bot{BOT_TOKEN}/setWebhook?url=

Success should be indicated by the following message:

{"ok":true,"result":true,"description":"Webhook was deleted"}

Outlook for the next part of the series

So far so good! For this part of the series, that’s it again!
We just learned how webhooks are working and how to enable your bot to make use of that method, how to analyze the data received by it on any webhook and how to disable it again, re-enabling getUpdate pull method.

Next time, I will show you how to prepare Heroku to host the Django app we will create in later chapters and already upload a first demo site to it.

I hope you enjoyed this part! Please let me know of all the things you either enjoyed or did not like that much and how I can do it better in the comments.

Marc Richter

Born in 1982, Marc Richter is an IT enthusiastic since 1994. He became addicted when he first put hands on their family’s pc and never stopped investigating and exploring new things since then.
He is married to Jennifer Richter and proud father of two wonderful children, Lotta and Linus.
His current professional focus is DevOps and Python development.

An exhaustive bio can be found at this blog post.

Found my articles useful? Maybe you would like to support my efforts and give me a tip then?

Der Beitrag Create your own Telegram bot with Django on Heroku – Part 4 – pull vs. push method erschien zuerst auf Marc Richter's personal site.

Briefly on Prototypes

{"key": "My house",

 "typeclass": "typeclasses.houses.MyHouse"}

• They allow you to customize individual objects easier. For example you could have a 'Goblin' base prototype with inheriting prototypes  Goblin Wizard" and "Goblin Chieftain" - all using the same Typeclass, but with different Attributes, equipment etc. 
• Prototypes can be manipulated and scripted by builders without needing full Python access. This means that while the Typeclasses are designed and supplied by the Python developer, the builders can then use that typeclass to make very different types of object instances in-game.

What's new

. Other actions: examine | delete Back (index) | Validate prototype | Quit 1: goblin_archer 5: goblin_archwizard 2: goblin_wizard 3: goblin 4: archwizard_mixin " />. Other actions: examine | delete Back (index) | Validate prototype | Quit 1: goblin_archer 5: goblin_archwizard 2: goblin_wizard 3: goblin 4: archwizard_mixin " border="0" data-original-height="430" data-original-width="1550" height="110" src="https://3.bp.blogspot.com/-tdauL-B6j1E/W3f01ltKlzI/AAAAAAAAJIs/Q5-cIY6AcGU_IXXasdzPWec7cxN061WrwCLcBGAs/s400/Screenshot%2Bfrom%2B2018-08-18%2B12-27-37.png" title="Prototype loading" width="400" />

| create prototype from object Back (index) | Quit 1: Griatch(#1)" /> | create prototype from object Back (index) | Quit 1: Griatch(#1)" border="0" data-original-height="392" data-original-width="1280" height="122" src="https://2.bp.blogspot.com/-XFm3KqhLBwE/W3f93upyVLI/AAAAAAAAJJA/eZDWGPHM93MWvt2T5W8Ytr4cHstGg8iXACLcBGAs/s400/Screenshot%2Bfrom%2B2018-08-18%2B13-04-48.png" title="Load prototype from object" width="400" />

Protfuncs

Strength = $randint(5, 20) 

| remove Back (index) | Validate prototype | Quit 1: Spawn in prototype's defined location (#2) 2: Spawn in Griatch's location (Limbo) 3: Spawn in Griatch's inventory 4: Update 2 existing objects with this prototype" /> | remove Back (index) | Validate prototype | Quit 1: Spawn in prototype's defined location (#2) 2: Spawn in Griatch's location (Limbo) 3: Spawn in Griatch's inventory 4: Update 2 existing objects with this prototype" border="0" data-original-height="1052" data-original-width="1554" height="270" src="https://1.bp.blogspot.com/-qHeRpEQTEzU/W3gFpJ9ge5I/AAAAAAAAJJw/-PVcelDk2CsIW1nJwWvMjazcJ9LyKx5pgCLcBGAs/s400/Screenshot%2Bfrom%2B2018-08-18%2B13-40-04.png" title="Spawning screen" width="400" />

Current status

Web Scraping has been of an interest to a lot of businesses and individuals with the immense potential of the quantitative data available online. The data collected can entice the growth of an organization or a personal business. Through this post, we will see through examples on how NodeJS can be used to scrape content […]

The post Web Scraping with NodeJS appeared first on The Tara Nights.

In this article, I will teach you exactly what a python lambda is. As a matter of fact if you know what functions are and how to define functions in Python then you already know what a lambda is. A Python lambda is just a Python function. But may be like a special type of [...]

The post Python Lambdas Explained (With Examples) appeared first on Afternerd.

If you are following me on Twitter, you have already seen a lot of (re)tweets related to Aadhaar. For the people first time hearing this term, it is a 12 digit unique identification number provided by the Unique Identification Authority of India (UIDAI). It is also the world’s largest bio-metric ID system. It is supposed to be a voluntary service.

From the very beginning, this project tried to hide the details from the Indian citizens. Let it be privacy advocates or security researchers or human rights activists, everyone predicted that this will become a monster, a mass surveillance system, a tool of choice of the power hungry dictators.

Like any other complex system, the majority of the people only see the advertisements from the government and completely miss all the problems and horror stories this project is creating. Here are a few links below for the interested people to read.

• IRC session log from Kiran Jonnalagadda
• Karana (detailed posts related to various parts of the project) Must read site.
• Aadhaar fail Details of scams/failures/deaths/other issues from Aadhaar project

Neither my wife, nor our daughter has an Aadhaar (I also don’t have one), that means Py (our daughter) did not get admission to any school last year.

Whenever security researchers or journalists tried to report on the project, the UIDAI tried to hide behind denials and police complaints against the journalists or researchers. There are various reports on how one can get access (both read/write) to the actual production database with as little as $10-30. We now have examples of terrorist organizations having access to the same database. The UIDAI kept telling how this is an unhackable technology and for security they have a 13 feet wall outside of the data center which in turn will keep all hackers away.

They have already build 360 degree databases on top of Aadhaar, and now they are trying to link DNA to the same system.

The current government of India tried their level best to argue in the Supreme Court of India to tell that Indians don’t have any rights to privacy. But, thankfully they failed in this effort, and the Supreme Court ruled privacy as a fundamental right. We are now waiting for the judgment on the Aadhaar (which will hopefully come out in the next few weeks).

Meanwhile, the evil nexus is pushing down Aadhaar to the throats of the Indian citizens and Pakistani spies and gods.

A few days ago, in an event in Jaipur, they asked Edward Snowden the following question.

How big of an issue is privacy?

The answer started with from where that argument comes from.

The answer is that Nazi Germany. The nazi minister of propaganda Joseph Goebbels did this. Because he was trying to change the conversation away from “What are your rights?” and “What evidences must the government show?” to violet them, to intrude into your private life and instead said “Why do you need your rights?”, “How can you justify your rights?”, “Isn’t strange that you are invoking your rights? Isn’t that unusual?”. But, in a free society this is the opposite of the way it is supposed to work. We don’t need to explain why you have a right. You don’t need to explain why it is valuable, why you need it. It is for the government to explain why you don’t deserve it. They go to a court, they show that you are a criminal. This is increasingly falling out of favor, because the governments and companies think that it is inefficient. It is too much work. Life would be easier, life would be more convenient for them, life would be more profitable for them if we didn’t have any rights at all.

But, privacy isn’t about something to hide, privacy is about something to protect. And that is the very concept of liberty. It is the idea that there can be some part of you, of your life, of your ideas that belong to you, not to society. And you get to make the decision about who you share that with. -- Edward Snowden

Why are we reading this in your blog?

This might a question for many of you. Why are reading this in a blog post or in a planet? Because we, the people with the knowledge of technology are also part of these evil plans. We now know about many private companies taking part with their local government to build 360 degree profiles, to track the citizens and to run the mass surveillance systems. For example, related to Aadhaar, for the last 4 years, Google silently pushed the Aadhaar support phone number (which now UIDAI is trying to stay away from) to every Google Android phone in India. When they got caught red handed, they claimed that they did it inadvertently. Finacle software by Infosys denies creation of bank accounts without Aadhaar. Microsoft is working to link Skype with Aadhaar. Bill Gates is trying to push the idea that Aadhaar is all good, and does not have any issues.

What can you do?

You can start by educating yourself first. Read more about the technologies which controls our lives. Have doubt about the things and try to understand how they actually work. Write about them, ask questions to the people in power. Talk about the issues to your friends and family.

This is not gong to be an easy task, but, we all should keep fighting back to make sure of a better future for our next generation.