In this chapter, we will continue to develop the cryptocurrency application by retrieving a number of top coins by their total volume across all markets in the last 24 hours.

We will perform these steps:

1. Create a button which will call the search coin function to retrieve the above mentioned data.
2. Create the search coin function to retrieve and to display the market data.

# get top 100 coins by their total volume across all markets in the last 24 hours
action_coin_volume = tk.Button(selectorFrame, text="Trade Volume", command=search_coin, state=DISABLED) # button used to get the top currency/cryptocurrency pair data
action_coin_volume.pack(side=LEFT)

def search_coin():  # display all market data of the currency/cryptocurrency pair

    sell_buy = ''
    try:
        currency_symbol = based.get()  # get the currency symbol
        url = "https://min-api.cryptocompare.com/data/top/totalvolfull"
        data = {'tsym': currency_symbol, 'limit': 100}
        r = requests.get(url, params=data)
        exchange_rate_s = json.loads(json.dumps(r.json()))

    except:
        print("An exception occurred")
    count = 0  # used to control new line for each symbol
    for value in exchange_rate_s['Data']:  # populate exchange rate string and the currency tuple
        for key, value1 in value.items():
            for key2, value2 in value1.items():
                if (type(value2) == dict):
                    for key3, value3 in value2.items():
                        if (key3 == "IMAGEURL"):
                            continue
                        sell_buy += str(key3) + " : " + str(value3) + "\n"
                    count += 1
                    if count == 2:
                        sell_buy += "\n"
                        count = 0

    text_widget.delete('1.0', END)  # clear all those previous text first
    s.set(sell_buy)
    text_widget.insert(INSERT, s.get())  # populate the text widget with new exchange rate data

The above function will retrieve the market data of the top 100 cryptocurrencies vs the selected currency within a period of 24 hours.

EUR vs top 100 cryptocurrency

Below are the steps to display those data:

1. Click on the Load data button to load the top currencies data to the currency’s combo box.
2. Select the currency which you wish to compare from the combo box, in this example we select EURO.
3. Click on the trade volume button to display the top 100 cryptocurrency market data vs the selected currency which is EUR0.

If you have missed out the previous chapter don’t forget to read them because the code in this chapter is interrelated with the previous one.

This is a blog post about trying to clean up the code that I have written for a feature that had merged and marked as experimental.

What did I do this week?

For the past few weeks, I have been working on an interesting feature called `unshelve --continue`. My mentor was actively giving reviews on the old patch even after getting that merged. So, I had to send a follow-up stack following the merged patch. I started with a patch[1] to modify the help text on verbose mode to say the user why this feature is still experimental and it's problems. Then, I sent a new stack:

[2]: D6679unshelve: store information about interactive mode in shelvedstate

[3]: D6687unshelve: create a matcher only if required on creating unshelve ctx

[4]: D6694unshelve: fix bug on a partial unshelve with --continue

[5]: D6697cmdutil: add allowunfinished to prevent checkunfinished() on docommit()

[6]: D6685unshelve: changes how date is set on interactive mode

[7]: D6686unshelve: handle stripping changesets on interactive mode

[8]: D6683unshelve: unify logic around creating an unshelve changeset

[9]: D6699tests: add test for unshelve --interactive --keep

What is coming up next?

Since I have been working on `unshelve --interactive` for a long time, my priority should be this only. I will try to clean the code and modify the tests to account for all cases possible. If that is getting over on time, I will work on an interesting issue[10].

Did you get stuck anywhere?

As usual, I got problems in rebasing and evolving changesets, I had to make the DAG linear after every time after resolving enormous conflicts. But, I got used to it and it wasn't much a problem per se.

Continue reading"New PyMOTW site logo"

1. What did you do this week?

This week I worked on the new Kurucz line list parser. Also, I finished the CMFGEN pipeline (merged PR #143).

NIST (PR #144) and Knox Long's recombination zeta (PR #145) are almost finished too.

2. What is coming up next?

I will continue working on the Kurucz parser.

3. Did you get stuck anywhere?

Kurucz parser is a really difficult one! It's taking me longer than I expected.

In this chapter, we will get a number of top 100 coins by their market cap with python. If you have read the previous chapter of this cryptocurrency project then you will find out that in this chapter we have only changed the GET’s url content a little bit to achieve the above-mentioned outcome. Below is the url we need to use to retrieve the above-mentioned outcome.

 url = "https://min-api.cryptocompare.com/data/top/mktcapfull"
 data = {'tsym': currency_symbol, 'limit': 100}

Besides that, we have also created a button to start the top_coin_by_cap method. Below is the full program up until now.

import json
from tkinter import *
import tkinter.ttk as tk
import requests
from Loader import Loader

win = Tk()  # Create tk instance
win.title("Crypto Calculator")  # Add a title
win.resizable(0, 0)  # Disable resizing the GUI
win.configure(background='white')  # change window background color

selectorFrame = Frame(win, background="white")  # create top frame to hold search button and combobox
selectorFrame.pack(anchor="nw", pady=2, padx=10)
currency_label = Label(selectorFrame, text="Select market / currency pair :", background="white")
currency_label.pack(anchor="w")

# create currency frame and text widget to display the incoming exchange rate data
s = StringVar()  # create string variable
currencyFrame = Frame(win)
currencyFrame.pack(side=TOP)
currency = Label(currencyFrame)
currency.pack()
text_widget = Text(currency, fg='white', background='black')
text_widget.pack()
s.set("Click the find button below to load the crypto currency - currency exchange rate from CCCAGG market")
text_widget.insert(END, s.get())

buttonFrame = Frame(win)  # create a bottom frame to hold the load button
buttonFrame.pack(side=BOTTOM, fill=X, pady=6)

# Create a combo box for crypto currency
crypto = tk.Combobox(buttonFrame)
crypto.pack(side=LEFT, padx=3)

# Create a combo box for markets
market = tk.Combobox(selectorFrame)
market.pack(side=LEFT, padx=3)

# create loader object then load the data into the crypto and market combo box
loader = Loader(crypto, market)
loader.load_data()

def top_coin_by_cap():  # Get a number of top 100 coins by their market cap

    sell_buy = ''
    try:
        currency_symbol = based.get()  # get the currency symbol
        url = "https://min-api.cryptocompare.com/data/top/mktcapfull"
        data = {'tsym': currency_symbol, 'limit': 100}
        r = requests.get(url, params=data)
        exchange_rate_s = json.loads(json.dumps(r.json()))

    except:
        print("An exception occurred")
    count = 0  # used to control new line for each symbol
    for value in exchange_rate_s['Data']:  # populate exchange rate string and the currency tuple
        for key, value1 in value.items():
            for key2, value2 in value1.items():
                if (type(value2) == dict):
                    for key3, value3 in value2.items():
                        if (key3 == "IMAGEURL"):
                            continue
                        sell_buy += str(key3) + " : " + str(value3) + "\n"
                    count += 1
                    if count == 2:
                        sell_buy += "\n"
                        count = 0

    text_widget.delete('1.0', END)  # clear all those previous text first
    s.set(sell_buy)
    text_widget.insert(INSERT, s.get())  # populate the text widget with new exchange rate data



def search_coin():  # display all market data of the currency/cryptocurrency pair

    sell_buy = ''
    try:
        currency_symbol = based.get()  # get the currency symbol
        url = "https://min-api.cryptocompare.com/data/top/totalvolfull"
        data = {'tsym': currency_symbol, 'limit': 100}
        r = requests.get(url, params=data)
        exchange_rate_s = json.loads(json.dumps(r.json()))

    except:
        print("An exception occurred")
    count = 0  # used to control new line for each symbol
    for value in exchange_rate_s['Data']:  # populate exchange rate string and the currency tuple
        for key, value1 in value.items():
            for key2, value2 in value1.items():
                if (type(value2) == dict):
                    for key3, value3 in value2.items():
                        if (key3 == "IMAGEURL"):
                            continue
                        sell_buy += str(key3) + " : " + str(value3) + "\n"
                    count += 1
                    if count == 2:
                        sell_buy += "\n"
                        count = 0

    text_widget.delete('1.0', END)  # clear all those previous text first
    s.set(sell_buy)
    text_widget.insert(INSERT, s.get())  # populate the text widget with new exchange rate data


def search_currency():  # display all market data of the cryptocurrency-currency pair

    sell_buy = ''
    try:
        base_crypto = crypto.get()  # get the desired crypto currency
        currency_symbol = based.get()  # get the currency symbol
        market_exchange = market.get()  # get the market
        url = "https://min-api.cryptocompare.com/data/generateAvg"
        data = {'fsym': base_crypto, 'tsym': currency_symbol, 'e': market_exchange}
        r = requests.get(url, params=data)
        exchange_rate_s = json.loads(json.dumps(r.json()))
    except:
        print("An exception occurred")

    for key, value in exchange_rate_s.items():  # populate exchange rate string and the currency tuple
        for key1, value1 in value.items():
            sell_buy += str(key1) + " : " + str(value1) + "\n"

    text_widget.delete('1.0', END)  # clear all those previous text first
    s.set(sell_buy)
    text_widget.insert(INSERT, s.get())  # populate the text widget with new exchange rate data

# Create currency combo box
base_currency = StringVar()  # create a string variable
based = tk.Combobox(selectorFrame, textvariable=base_currency)
based.pack(side=LEFT, padx=3)

action_search = tk.Button(selectorFrame, text="Search", command=search_currency,
                          state=DISABLED)  # button used to get the cryptocurrency/currency pair exchange rate
action_search.pack(side=LEFT)

# get top 100 coins by their total volume across all markets in the last 24 hours
action_coin_volume = tk.Button(selectorFrame, text="Trade Volume", command=search_coin, state=DISABLED)  # button used to get the top currency/cryptocurrency pair data
action_coin_volume.pack(side=LEFT)

# get top 100 coins by their market cap
action_coin_market_cap = tk.Button(selectorFrame, text="Market Cap", command=top_coin_by_cap, state=DISABLED)  # button used to get the top currency/cryptocurrency pair data
action_coin_market_cap.pack(side=LEFT)

def get_exchange_rate():  # this method will display the incoming exchange rate data after the api called

    global exchange_rate
    global base_crypto

    # read the currency file
    c_string = ''
    with open('currency.txt') as fp:
        for currency in fp.readlines():
            c_string += currency[:-1] + ","
    c_string = c_string[:-1]

    base_crypto = crypto.get()  # get the desired crypto currency
    try:
        url = "https://min-api.cryptocompare.com/data/price"  # url for API call
        data = {'fsym': base_crypto, 'tsyms': c_string}
        r = requests.get(url, params=data)
        exchange_rate_s = json.loads(json.dumps(r.json()))

    except:
        print("An exception occurred")

    curr1 = tuple()  # the tuple which will be populated by currency
    sell_buy = ''

    for key, value in exchange_rate_s.items():  # populate exchange rate string and the currency tuple
        sell_buy += base_crypto + ":" + key + "  " + str(value) + "\n"
        curr1 += (key,)

    # fill up combo boxes for both currency and cryptocurrency
    based['values'] = curr1
    based.current(0)

    text_widget.delete('1.0', END)  # clear all those previous text first
    s.set(sell_buy)
    text_widget.insert(INSERT, s.get())  # populate the text widget with new exchange rate data

    # enable all buttons
    action_search.config(state=NORMAL)
    action_coin_volume.config(state=NORMAL)
    action_coin_market_cap.config(state=NORMAL)


action_vid = tk.Button(buttonFrame, text="Load",
                       command=get_exchange_rate)  # button used to load the exchange rate of currency pairs
action_vid.pack()

win.iconbitmap(r'ico.ico')
win.mainloop()

EUR vs top cryptocurrency

This project will continue for a while and we might rearrange the button and combo box in the future to suit the layout of the user interface.

Python is great, but pure Python code sometimes has one problem: It’s slow.

Fortunately there are several great solutions to improve the performance, like Numpy, Cython, Numba, Pypy.

All of the above solutions have different drawbacks:

Numpy and Numba are big modules. In addition Numpy is not always fast enough. Pypy is not 100% compatible and a heavy solution if it is used in addition to CPython. Cython is complex and you need a C-compiler. Recently I’ve found one more solution which maybe is not that well known: Lua integration in Python programs.

Lua is another scripting language with dynamic data types.

So I asked myself:

Does it make sense to have another scripting language inside Python scripts?

Let’s have a look at a simple example: Mandelbrot

First of all the pure Python example:

fromnumpyimportcomplex,arangedefPyMandelbrotIter(c):z=0foritersinrange(200):ifabs(z)>=2:returnitersz=z**2+creturnitersdefPyMandelbrot(size):image=Image.new('RGB',(size,size))pix=image.load()t1=time.time()xPts=arange(-1.5,0.5,2.0/size)yPts=arange(-1,1,2.0/size)forxx,xinenumerate(xPts):foryy,yinenumerate(yPts):pix[xx,yy]=PyMandelbrotIter(complex(x,y))dt=time.time()-t1print(f"dt={dt:.2f}")image.show()

Runtimes of this example on a Core i7 laptop with Python 3.7 and Windows 10:

Now the Lua example integrated in a Python script:

fromlupaimportLuaRuntimelua_code='''\
function(N, i, total)
  local char, unpack = string.char, table.unpack
  local result = ""
  local M, ba, bb, buf = 2/N, 2^(N%8+1)-1, 2^(8-N%8), {}
  local start_line, end_line = N/total * (i-1), N/total * i - 1
  for y=start_line,end_line do
    local Ci, b, p = y*M-1, 1, 0
    for x=0,N-1 do
      local Cr = x*M-1.5
      local Zr, Zi, Zrq, Ziq = Cr, Ci, Cr*Cr, Ci*Ci
      b = b + b
      for i=1,49 do
        Zi = Zr*Zi*2 + Ci
        Zr = Zrq-Ziq + Cr
        Ziq = Zi*Zi
        Zrq = Zr*Zr
        if Zrq+Ziq > 4.0 then b = b + 1; break; end
      end
      if b >= 256 then p = p + 1; buf[p] = 511 - b; b = 1; end
    end
    if b ~= 1 then p = p + 1; buf[p] = (ba-b)*bb; end
      result = result .. char(unpack(buf, 1, p))
    end
    return result
end
'''defLuaMandelbrot(thrCnt,size):defLuaMandelbrotFunc(i,lua_func):results[i]=lua_func(size,i+1,thrCnt)t1=time.time()lua_funcs=[LuaRuntime(encoding=None).eval(lua_code)for_inrange(thrCnt)]results=[None]*thrCntthreads=[threading.Thread(target=LuaMandelbrotFunc,args=(i,lua_func))fori,lua_funcinenumerate(lua_funcs)]forthreadinthreads:thread.start()forthreadinthreads:thread.join()result_buffer=b''.join(results)dt=time.time()-t1print(f"dt={dt:.2f}")image=Image.frombytes('1',(size,size),result_buffer)image.show()

Runtimes of this example on a performance laptop with Python 3.7 and Windows 10:

The above results are very impressive. As you can see Lua is really much faster. And as you can also see: You can parallelize with threads!

The module lupa, which comes with a Lua interpreter and a JIT compiler, is a very interesting alternative for speeding up long running tasks.

The Lua solution has the following advantages:

• The lupa module is very small.
• Lua is much faster than Python.
• You can run Lua scripts in parallel with threads.
• Lua is very easy to read and code.
• You can easily integrate Lua scripts in Python code.
• You can easily access Python objects within Lua and vice versa.
• There are many extension modules available for Lua (~2600, see luarocks.org).

Give lupa a try. It’s easy to use and really great!

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2019-07-28 11:32:11 GMT

1. Is Python `list.extend(iterator)` guaranteed to be lazy? - [11/2]
2. Count the identical pairs in two lists - [10/6]
3. Get the next element of list in Python - [8/4]
4. Infer generic lambda parameters from another generic lambda's parameters - [8/1]
5. Group recursively adjacent tuples from a list in Python - [7/5]
6. Significance of Capitalization in Python collections module? - [7/1]
7. How to remove consecutive identical words from a string in python - [6/7]
8. Random list only with -1 and 1 - [6/6]
9. How to align two lists of numbers - [6/5]
10. Structuring a 2D array from a pandas dataframe - [6/2]

Hi all. Hope we all passed evaluation 2 and our work continues!

What did you do this week?

The text PR saw a couple of updates, such as support for more ways to specify certain arguments (take more types), a change to the underlying method used for 1D array resizing (replaced scipy.signal with scipy.interpolate), and unit tests for text helpers.

I moved the code that could be reused by different neural network packages (Keras, PyTorch, Tensorflow, etc) into a new ‘nn’ package.

A major feature was adding PyTorch support in a new branch off the text PR. I got working support for images (torchvision and reusable numpy code helped a lot), but text support is yet to be added.

What is coming up next?

Work will continue as usual: Text PR needs fixes, docs and tests. In particular, I will need to train better models for integration tests.

I should open a PyTorch WIP PR once text support is added. That includes obtaining a text model with which I can do manual tests. A stretch goal would be to also get started with docs and other things that come with PR’s.

Since one of my mentors is back from holidays, I will take any feedback, and we can also plan out the last 3 weeks of GSoC. Besides the current tasks, there is some room in the schedule for other things.

Did you get stuck anywhere?

As mentioned, I need to train text models specifically designed for integration tests – small, covering many cases (kinds of layers used, character/word level tokenization, output classes, etc), and working well with explanations. I tried to add an integration test but the model I used gave dubious explanations for simple sentences, so changing the model is a solution I can try.

I was keen on opening a PyTorch PR this week, but a blocker was adding text support. Specifically I didn’t have an off-the-shelf pretrained text model to test with. The simplest solution I tried was saving a text classifier from a public kaggle notebook (I got an error when trying to save so I will need to retry).

I shall see you next week!

Tomas Baltrunas

Greetings everyone! Update for this week is as follows. In this phase, I sent the remaining patches required to complete hg continue support for core Mercurial repository.

What did I do this week?
As stated in the previous week I sent patches for hg continue for for graft, transplant and histedit.[1][2][3][4]
I also created logic for hg transplant --abort[5] as asked by @Pulkit and added its support to hg abort plan [6]. These patches are still to be merged and under review.

What is coming up next?

This week I am planning to complete support for hg continue for evolve. Also, I will start my work on logic for hg update --abort which is one of the most demanded features for Mercurial.
My patches for hg abort for evolve merged as they still left to get reviews as most of the people in the community are busy with the new version release.

Did you get stuck anywhere?

I did get stuck when I started with the logic for aborting a transplant but @Pulkit again helped me out of the situation and suggested that I should look into the recover method of the transplanter class.

What did you do this week?

Hoorya!! I passed the second evaluation. In the evaluation week I mainly completed the only test which is left before the second evaluation i.e Adding test for the ploneGenerator method which generate all the Gatsby Node initially.  It is the most important generator function. I never tested the generator function in my life. I searched what is the best way to test the generator function and I found some response and successfully Send a pr for that. After that there is simple bug i.e Foreach array method randomly calling the api but we need syncrounus behaviour. After devoting one day and lot of thinking I found that only replacing the foreach method with for loop does the trick. Then I successfully created the pr for that also. You can see the pr link below.

https://github.com/collective/gatsby-source-plone/pull/218 

https://github.com/collective/gatsby-source-plone/pull/225

What is coming up next?

Upto this time I have successfully implemented the MVP for my project. Written test for all the function which is present in the source code. Now, I only have to refactor the code and write documentation for the added feature. So, You can think that this period mainly deal with refactoring and documenting.

Did you get stuck anywhere?

Really for first time in this project I suffer with simple issue which is converting the foreach method to for loop. It was not easy to find what's going wrong with foreach. Why it doesn't behaviour synchronously. But after lot of thinking and experiment I am able to attend the behaviour i.e synchronous.

So, what I have done in second period. For me the answer is very simple, MVP is completed in first phase which give me ample time to write test for the added feature and other function which is not tested earlier. In second Period I read lot of documentation of Jest. I have gone through each Matchers which is Provided by jest. Spent almost a week on reading it. After reading the documentation I feel very comfortable in writing test in jest. From my experience I would say don't dive in coding without knowing anything about the subject. First read the documentation of the following framework and become comfortable with the language. After that you will able to solve the problem in very less time. I have written around 10 test in this period which covers every aspect of the source code. I got stuck at the point where I have to write test for the generator function. I have to mock lot of data because It is the main function of source code which generates all the gatsby nodes initially. I also make some mistake. I am fetching data from the Plone CMS and then doing logging. I just forget that console log not print the nested array. So, I have made a huge mistake using console log. After lot of try I am not able to write test properly. I stuck in loop fixing one thing break another, after lot of afford I am able to solve this problem at 3am when I am going to bed after hectic day. I really enjoyed this phase a lot, I learned lot of new things and implemented it. I am very happy with myself what I am doing right now. In third phase I mainly write documentation and refactoring the code into small pieces of function so that It can enhance the  readability of code.

If You find any thing interesting and anything which you want to ask, I will be happy to answer it :)

Thank You

Hi and welcome again!

Let's get going.

What did you do this week?

Last week I was fixing bugs and errors in the landing page code. I finished with that(almost). I had a really hard time, because I created a lot more errors while fixing a few errors. But Finally, I don't remember how but I fixed those. Well, I still need to do a few more adjustments but its fine. Also, I wrote the code for the cheatsheet and its almost ready.

What's coming next?

After I finish with the Landing page and cheatsheet, I will create a Mockup for the documentation page of EOS and later I may have to implement it too. Those are some later plans.

Did you stuck anywhere?

Yes, yes I was stuck for a long time here, when I was fixing the landing page. Man, I don't know how but some vs-code extension played with my code and created some .map files on its own. And keeps on generating those files automatically every time I use gulp. That was a pain but eventually, I was able to fix that!

In this chapter, we will create a button which will call a function to retrieve the top 50 exchanges and the trading volume for a cryptocurrency/currency pair. The API needs the selected cryptocurrency and the selected currency symbol to make the call.

BTC vs EURO for various exchanges

First we will create the exchange button.

# get top 50 exchanges by volume for currency/cryptocurrency pairs
action_coin_top_exchange = tk.Button(selectorFrame, text="Top Exchange", command=top_exchange, state=DISABLED)  # button used to get the top currency/cryptocurrency pair data
action_coin_top_exchange.pack(side=LEFT)

Then we create the function.

def top_exchange():  # Get a number of top 50 exchanges by their volumes

    sell_buy = ''
    try:
        currency_symbol = based.get()  # get the currency symbol
        crypto_symbol = crypto.get() # get the cryptocurrency symbol
        url = "https://min-api.cryptocompare.com/data/top/exchanges"
        data = {'tsym': currency_symbol, 'fsym': crypto_symbol, 'limit': 50}
        r = requests.get(url, params=data)
        exchange_rate_s = json.loads(json.dumps(r.json()))

    except:
        print("An exception occurred")

    for value in exchange_rate_s['Data']:  # populate exchange rate string
        for key, value1 in value.items():
            sell_buy += str(key) + " : " + str(value1) + "\n"

        sell_buy += "\n"


    text_widget.delete('1.0', END)  # clear all those previous text first
    s.set(sell_buy)
    text_widget.insert(INSERT, s.get())  # populate the text widget with new exchange rate data

If you miss out the first few posts regarding this project, you can read them all on this website under the python category.

Sops is a secret management tool from Mozilla. According to the official Github page, it is defined as:

sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP.

In this blog post, I am going to show you how you can use it along with GPG to encrypt secrets in YAML files for your Ansible roles/playbooks.

Installation

Download the official binary from the release page, or you can build and install from the source code.

Basic usage (manual)

First let us create a .sops.yaml in the git repository root directory. In this example, I am saying that I can encrypt any .yml using the two given GPG fingerprints.

creation_rules:
  # If vars file matchs "*.yml", then encrypt it more permissively.
    - path_regex: \.yml$
      pgp: "A85FF376759C994A8A1168D8D8219C8C43F6C5E1,2871635BE3B4E5C04F02B848C353BFE051D06C33"

Now to encrypt a file in place, I can use the following command:

sops -e -i host_vars/stg.dgplug.org.yml

If we open the file afterward, we will see something like below.

mysql_root_password: ENC[AES256_GCM,data:732TA7ps+qE=,iv:3azuZg4tqsLfe5IHDLJDKSUHmVk2c0g1Nl+oIcKOXRw=,tag:yD8iwmxmENww+waTs5Kzxw==,type:str]
mysql_user_password: ENC[AES256_GCM,data:YXBpO54=,iv:fQYATEWw4pv4lW5Ht8xiaBCliat8xdje5qdmb0Sff4Y=,tag:cncwg2Ops35l0lWegCSEJQ==,type:str]
mysql_user: ENC[AES256_GCM,data:cq/VgDlpRBxuHKM+cw==,iv:K+v6fkCIucMrMJ7pDRxFS/aHh0OCxqUcLJhZIgCsfA0=,tag:BD7l662OVOWRaHi2Rtw37g==,type:str]
mysql_db_name: ENC[AES256_GCM,data:hCgrKmU=,iv:/jnypeWdqUbIRy75q7OIODgZnaDpR3oTa0G/L8MRiZA=,tag:0k6cGNoDajUuKpKzrwQBaw==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    lastmodified: '2019-07-29T04:05:09Z'
    mac: ENC[AES256_GCM,data:qp9yV3qj0tYf/EaO0Q3JdlpPvm5WY4ev1zGCVNoo+Anm/esj0WHlR7M7SNg54xRTUwMhRRnirx7IsEC8EZW1lE+8DObnskemcXm93CJOBfVzQOX/RvCSR4rMp2FgBEPZADCDiba1X2K/9myU96lADME0nkdmX9YjhOLMFJ6ll4o=,iv:2WNKhl81FI/qw6mRKpK5wRYjqK16q1ASeCJYpEeuhj4=,tag:v4PlGT4ZmPUxj7aYIendVg==,type:str]
    pgp:
    -   created_at: '2019-07-29T04:05:06Z'
        enc: |-
            -----BEGIN PGP MESSAGE-----


            wcFMA/uCql0ybaddARAADkHqV/mEgVoTxVHkRKW3mjajReKaQ5Mz/VwMal3GsjKr
            8aGnghCjtgJE01wCBjrTfNKKmlf5JjMSFufy9pG0xE+OFOXt+pnJFDSro26QnPFG
            VlvkvnjTxw4gV/mIvxUXTT6rmijvQSHMXdyEGmyHu3kNprKuuN37xZ4SYSWg7pdO
            vee4DSOaw+XfdgYUF+YSEjKMZ+qevRtzJL4hg9SZvEsHMQObueMBywAc5pyR6LvS
            ZuAS5SS+cPXnyMJLemqfU7L+XMw2NMrmNYFXOWnMMny1Hez5fcmebJp+wjDqWJTX
            j9vJvXLuNAglFvL1yz2rNJYTb0mC9chLT7YxEa9Z9JHWezUB8ZYuOC6vRf18/Hz8
            e6Gd2sncl4rleCxvKZF9qECsmFzs4p7H5M+O31jnjWdnPBD8a84Du3wdeWiI5cRF
            d7/aUXEdGJQy7OVbzGE1alDOSIyDD2S73ou2du7s/79Wb11RwQV898OyGgmjWm0Y
            7hTsBiBXnayQdjtg6qKlvoWIn79PU0YmDYLujMiXDQPJLV3ta82dcK2B1yTCLuSO
            nGfFzNOSNemmniyEuMI16SrfYDsf9l/K3gec+TRNvEdc1GqO4gFblQWptPE7KIIC
            oBVMDLUJSpOf5yF7Izedy5wBb8ZmzJAvpchvTMUls2+En4ifYh90cauXxiP6edPS
            4AHkZMebg44aCefn4zMdKdUhbOFxbuA/4I3hQWLgmuJMFlcI4Orl5tsRjXwCfQ9S
            jOdGAUJ8usV9gT4IXj73WfN1JJHj7DTgoORXFtDMs2Yy/rPPR4H9msSL4reJGZLh
            dEAA
            =LoMY
            -----END PGP MESSAGE-----
        fp: A85FF376759C994A8A1168D8D8219C8C43F6C5E1
    -   created_at: '2019-07-29T04:05:06Z'
        enc: |-
            -----BEGIN PGP MESSAGE-----


            wcFMA0HZfc7pf7hDARAAX6xF6qP9KQJ5wLn0kv5WVf8HgOkk+2ziIuBH411hVEir
            oN4bwwnL+DEYZm2aFvZl5zQElua7nGkQK041DecPbOCCBqThv3QKVMy2jScG5tTj
            jxGgh8W/KxdwIY7teRaCRNDkT18IhtBc4SO2smJEtPeGIptvDHLjETBFbDnZeo6/
            PG4QIA1Rfvm14n1NR56oibWduwvb1wrm4tGYPDx8mVgfugxeIhoqlZ87VrivrN+2
            40S/rwWQ/aEVM1A8q19DGkXYVBcxQA1dGBrKLDPtiCq/LCSDNY4iuzMjzQuHPjgM
            bS0lWv8fvSp6iIZlB3eSRPW9Ia8tRJpEfTLS8jiHlcCZ4Vy3fW6EijBf00iSy5hP
            Y54TCERscXt1/UOW2ACYTNhMfuiO+WuG6Vjdns1NsSUVazqxmf+kBMwl06/HyUwL
            KAYTOB2hipYUm6mlpSBfDgBKjQq8dgvxlWP0Ay0of0p4ZzFv7MepYkJA+gwb0hUt
            rui9GVE/uys8W8buYqfM2ABzIG4GrH3rELh8eW8oPwlu7rgS7YGhyog2xabJyQnj
            BZ65wwu5TzMq+n5v1+878teOzqqD/F+6X5dw0jF95bDHKdA64JR/Uxlj75sp59GH
            e/+3UDm0UwSILDrYJkcVaTnrt3wPjQAw4ynKZuN+k6KvmDCkGquHNaM+2hqvWq/S
            4AHkgpZHzmU14QmfJy7RN2HPduHhSeCh4LrhMzngJuJyH72G4IPlP4WwPJUhrKMt
            UI+Azl61rIZEm6n82oWbY1gIrrIygWLg1OSD/0Ly6KbO4/W1NipWU53w4nAo7abh
            3gkA
            =YXu1
            -----END PGP MESSAGE-----
        fp: 2871635BE3B4E5C04F02B848C353BFE051D06C33
    unencrypted_suffix: _unencrypted
    version: 3.1.1

If you look closely, we can see that sops only encrypts the values, not the keys in the YAML file.

Decrypting as required on runtime

We can use a small Python script to enable runtime decryption of the file as required in Ansible. Create a vars_plugin directory in the git repository root, and then put the following code in there as sops.py.

from __future__ import (absolute_import, division, print_function)
__metaclass__ = type


DOCUMENTATION = '''
    vars: sops
    version_added: "N/A"
    short_description: In charge of loading SOPS-encrypted vars
    description:
        - Loads SOPS-encrytped YAML vars into corresponding groups/hosts in group_vars/ and host_vars/ directories.
        - Only SOPS-encrypted vars files, with a top-level "sops" key, will be loaded.
        - Extends host/group vars logic from Ansible core.
    notes:
        - SOPS binary must be on path (missing will raise exception).
        - Only supports YAML vars files (JSON files will raise exception).
        - Only host/group vars are supported, other files will not be parsed.
    options: []
'''


import os
import subprocess
import yaml
from ansible.errors import AnsibleError, AnsibleParserError
from ansible.module_utils._text import to_bytes, to_native, to_text
from ansible.inventory.host import Host
from ansible.inventory.group import Group
from ansible.utils.vars import combine_vars


FOUND = {}


# Import host_group_vars logic for file-walking functions.
# We'll still need to copy/paste and modify the `get_vars` function
# and edit below to insert a call to sops cli.
from ansible.plugins.vars.host_group_vars import VarsModule as HostGroupVarsModule


# All SOPS-encrypted vars files will have a top-level key called "sops".
# In order to determine whether a file is SOPS-encrypted, let's inspect
# such a key if it is found, and expect the following subkeys.
SOPS_EXPECTED_SUBKEYS = [
    "lastmodified",
    "mac",
    "version",
]



class AnsibleSopsError(AnsibleError):
    pass



class VarsModule(HostGroupVarsModule):


    def get_vars(self, loader, path, entities, cache=True):
        """
        Parses the inventory file and assembles host/group vars.


        Lifted verbatim from ansible.plugins.vars.host_group_vars, with a single
        in-line edit to support calling out to the SOPS CLI for decryption.
        Only SOPS-encrypted files will be handled.
        """


        if not isinstance(entities, list):
            entities = [entities]


        super(VarsModule, self).get_vars(loader, path, entities)


        data = {}
        for entity in entities:
            if isinstance(entity, Host):
                subdir = 'host_vars'
            elif isinstance(entity, Group):
                subdir = 'group_vars'
            else:
                raise AnsibleParserError("Supplied entity must be Host or Group, got %s instead" % (type(entity)))


            # avoid 'chroot' type inventory hostnames /path/to/chroot
            if not entity.name.startswith(os.path.sep):
                try:
                    found_files = []
                    # load vars
                    b_opath = os.path.realpath(to_bytes(os.path.join(self._basedir, subdir)))
                    opath = to_text(b_opath)
                    key = '%s.%s' % (entity.name, opath)
                    if cache and key in FOUND:
                        found_files = FOUND[key]
                    else:
                        # no need to do much if path does not exist for basedir
                        if os.path.exists(b_opath):
                            if os.path.isdir(b_opath):
                                self._display.debug("\tprocessing dir %s" % opath)
                                found_files = loader.find_vars_files(opath, entity.name)
                                FOUND[key] = found_files
                            else:
                                self._display.warning("Found %s that is not a directory, skipping: %s" % (subdir, opath))


                    for found in found_files:
                        # BEGIN SOPS-specific logic
                        if self._is_encrypted_sops_file(found):
                            new_data = self._decrypt_sops_file(found)


                            if new_data:  # ignore empty files
                                data = combine_vars(data, new_data)
                        # END SOPS-specific logic


                except Exception as e:
                    raise AnsibleParserError(to_native(e))
        return data


    def _is_encrypted_sops_file(self, path):
        """
        Check whether given filename is likely a SOPS-encrypted vars file.
        Determined by presence of top-level 'sops' key in vars file.


        Assumes file is YAML. Does not support JSON files.
        """
        is_sops_file_result = False
        with open(path, 'r') as f:
            y = yaml.safe_load(f)
            if type(y) == dict:
                # All SOPS-encrypted vars files will have top-level "sops" key.
                if 'sops' in y.keys() and type(y['sops'] == dict):
                    if all(k in y['sops'].keys() for k in SOPS_EXPECTED_SUBKEYS):
                        is_sops_file_result = True
            return is_sops_file_result


    def _decrypt_sops_file(self, path):
        """
        Shells out to `sops` binary and reads decrypted vars from stdout.
        Passes back dict to vars loader.


        Assumes that a file is a valid SOPS-encrypted file. Use function
        `is_encrypted_sops_file` to check.


        Assumes file is YAML. Does not support JSON files.
        """
        cmd = ["sops", "--input-type", "yaml", "--decrypt", path]
        real_yaml = None
        try:
            decrypted_yaml = subprocess.check_output(cmd)
        except OSError:
            msg = "Failed to call SOPS to decrypt file at {}".format(path)
            msg += ", ensure sops is installed in PATH."
            raise AnsibleSopsError(msg)
        except subprocess.CalledProcessError:
            msg = "Failed to decrypt SOPS file at {}".format(path)
            raise AnsibleSopsError(msg)
        try:
            real_yaml = yaml.safe_load(decrypted_yaml)
        except yaml.parser.ParserError:
            msg = "Failed to parse YAML from decrypted SOPS file at {},".format(path)
            msg += " confirm file is YAML format."
            raise AnsibleSopsError(msg)
        return real_yaml

From the next you will try to use any of the encrypted vars files in an Ansible run, it will ask for the GPG passphrase to decrypt the file as required.

Thank you Conor Schaefer for the original version of the Python script.

The post PyDev of the Week: Ines Montani appeared first on The Mouse Vs. The Python.