Confluent Hub Client

Confluent Hub is an online repository for extensions and components for Kafka. Kafka is based on extensible model for many of its services. 

It allows plug-ins and extensions which makes it generic enough to be suitable for many real world streaming based applications. They include both Confluent and 3rd party components. 

Generally, you would go there to search for components including:

• Connectors
• SMT (Single Message Transforms)
• Converters

Installing Connectors from Confluent Hub:

The enterprise version of Confluent provides a script for installing Connectors and other components from Confluent Hub (the script is not included in the Open Source version).

If we're using the enterprise version, we can install a connector using the following command:

$CONFLUENT_HOME/bin/confluent-hub install confluentinc/kafka-connect-mqtt:1.0.0-preview

More no of connectors can be found under below link:

https://www.confluent.io/product/connectors/

Access Bitbucket using python

Bitbucket is a Git-based source code repository hosting service owned by Atlassian.

Bitbucket Server is a combination Git server and web interface product written in Java and built with Apache Maven.

It allows users to do basic Git operations  while controlling read and write access to the code. It also provides integration with other Atlassian tools.

Now a days for reporting and for ETL operations, demand for python is increasing.

Using Python request library we can access bitbucket.

Below is sample code snippet.

import requests

import json

import pandas as pd
import io
url = 'https://api.bitbucket.org/2.0/repositories/Abcd'
headers = {'Content-Type': 'application/json'}
USERNAME = 'xxxxxx'
PASSWORD = 'yyyyyy'
response = requests.get(url, auth=(USERNAME, PASSWORD), headers=headers)
if response.status_code != 200:
    print('Status:', response.status_code, 'Headers:', response.headers, 

'Error Response:', response.json())
    exit()

df = pd.read_json(io.StringIO(response.text))
# j = a.assigng().assigng()

dk = pd.json_normalize(df['values'])
dk.to_excel('Bitbucket_INC_Report.xlsx', sheet_name='SLA_Report', index='False')

Install and Use Gremlin in a Docker Container

Gremlin is a simple, safe and secure way to use Chaos Engineering to improve system resilience. You can use Gremlin with Docker in a variety of ways.

It is possible to attack Docker containers and it is also possible to run Gremlin in a container to create attacks against the host or other containers.

• Create a Gremlin account: https://www.gremlin.com/demo/

• Login to the Gremlin App using your Company name and sign-on

credentials.

• Identify “TeamID” and “Secret Key” by navigating to

Settings>>TeamSettings>>Configuration

• Issue below command to in docker to pull the official Gremlin Docker image

and run the Gremlin daemon.

docker run -d --net=host \

--cap-add=NET_ADMIN --cap-add=SYS_BOOT --cap-add=SYS_TIME \

--cap-add=KILL \

-v $PWD/var/lib/gremlin:/var/lib/gremlin \

-v $PWD/var/log/gremlin:/var/log/gremlin \

-v /var/run/docker.sock:/var/run/docker.sock \

-e GREMLIN_TEAM_ID="$GREMLIN_TEAM_ID" \

-e GREMLIN_TEAM_SECRET="$GREMLIN_TEAM_SECRET" \

gremlin/gremlin daemon

• Use docker ps to see all running Docker containers:

       sudo docker ps

• Jump into your Gremlin container with an interactive shell

     sudo docker exec -it <gremlin container_id> echo “Running”

• From within the container, check out the available attack types:

      gremlin help attack-container

Microsoft Restricted Free Usage of Azure DevOps Pipeline

 Microsoft has changed the policy to allow the free tier of a hosted agent pools for public and private projects of newly created DevOps organizations by citing the reason that many are abusing this feature by sending a huge amount of traffic on these hosted agents pools. 

Due to this reason, many are getting the following error during the build pipeline.

No hosted parallelism has been purchased or granted. To request a free parallelism grant, please reach out to azpipelines-freetier@microsoft.com with your name and organization name.

Microsoft comes with the some temporary alternative approach until the automate is implemented to grant the permission for those users who requires the free hosted agent pools. 

Now users needs to be drop the email to get the free tier access based on the project visibility types which is explained below

Private Project

To get the free tier access of Microsoft hosted agent pools for private projects, send the email to azpipelines-freetier@microsoft.com with the following details.

• Your Name
• Microsoft azure DevOps organization name (dev.azure.com/yourorganization)

Public Project

To get the free tier access of Microsoft hosted agent pools for public projects, send the email to azpipelines-ossgrant@microsoft.com with the following details.

• Your Name
• Microsoft azure DevOps organization name (dev.azure.com/yourorganization)
• Link of the repositories you want to build
• Short description of your project.

Youtube Video: https://www.youtube.com/watch?v=WiHxt5YnWYs

What is Azure Devops?

 Azure DevOps is also known as Microsoft visual studio team services (VSTS) is a set of collaborative development tools built for the cloud. 

VSTS was commonly used as a standalone term, and Azure DevOps is a platform which is made up of a few different products, such as:

• Azure Test Plans
• Azure Boards
• Azure Repos
• Azure Pipeline
• Azure Artifacts

The azure pipeline is the CI component of azure DevOps. 

An azure pipeline can connect to any number of source code repositories such as Azure Repos, GitHub, Tests, to grab code and artifacts for application delivery.

Azure Boards:

Azure board is basically used for planning, tracking for Work Items, Backlog, and Sprints. Azure Board is also called an ALM (application life cycle management) tool just like other ALM tools JIRA, SpiraTeam, etc. Inside the Azure board, we can create workflows, issue types, epic, and many other components of ALM.

Azure Repos:
Azure Repos is a place where we can create, manage and store versions of our codebase. We can use it for code review and other version control functionality.

Azure Repos basically provides us two types of version control systems now GIT (Distributed version control) and TFVC (centralized version control).

Azure pipeline:

Azure pipeline is one of the automation processes of DevOps which allows us to create a build, release, test, and deploy on the target machine. We can say in simple words Azure pipeline is the combination CI/CD.

Azure Artifacts:

Azure Artifacts is one of the extensions of Azure DevOps which helps us to create, host, manage and share packages across the team. Azure Artifacts supports multiple types of packages e.g NPM, Nuget, Maven, Python, etc.. Azure Artifacts are basically a collection/ output of dll, rpm, jar and many other types of files. One of those extension based files may contain metadata.

Azure test Plans:

Azure test plans support advance test management solutions for customer feedback, UAT, Automation testing, and manual testing.

Tasks That Can be performed on Azure Devops:

What can Azure DevOps do?

• Planning
  As discussed above, the Azure board is an ALM tool. So Azure board helps us to maintain an Agile process which means Azure DevOps helps us to plan, track development history for individual developers and detects issues using a kanban and scrum. 
  
  
• Develop
  To manage our workspace and code repository multiple types of version control are already integrated with Azure DevOps. Almost all features related DevOps already inbuilt inside.
  
  
• Integration/Build
  We can use continuous integration and continuous deployment (CI/CD) with the help of the Azure pipeline. There are multiple types of job agents, and templates are already available inside. 
  
  
• Package
  As we know we have already an inbuilt extension available, Azure Artifacts, so we don't need to go anywhere for any type of packages.
  
  
• Testing
  As we know Testing is an important part of the DevOps process. So Azure DevOps provides multiple types of inbuilt testing templates and management tools.
  
  
• Release
  As we know the software release process is a very critical moment of DevOps. So Azure DevOps has already multiple types of inbuilt features to make the process easy and risk-free.
  
  
• Configure/Operate
  As per my experience, it's very easy to configure and operate, because Azure DevOps has a simple process to configure organization, kanban, sprints, etc..
  
  
• Monitor
  Each phase of the DevOps life cycle should be monitor because it helps to ensure the health, performance, and reliability of your application.
  
  
• Reporting
  Azure DevOps has a dashboard where we can see reports, a summary and can generate an analytical report with BI integration. 
  
  
• Wiki
  Azure DevOps Wiki helps us to distribute information, share knowledge and collaborate across team and stakeholder. A Wiki can be used to explain about the project's user stories, sprints, release notes, etc..

Link: https://www.youtube.com/watch?v=WiHxt5YnWYs

Dockerfile

A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image

The Docker build command executes the Dockerfile and builds a Docker image from it.

 A Docker image typically consists of:

• A base Docker image on top of which to build your own Docker image.
• A set of tools and applications to be installed in the Docker image.
• A set of files to be copied into the Docker image (e.g configuration files).
• Possibly a network (TCP / UDP) port (or more) to be opened for traffic in the firewall..etc

Dockerfile Structure

A Dockerfile consists of a set of instructions. Each instruction consists of a command followed by arguments to that command, similar to command line executables. 

Here is a simple example of a Dockerfile:

# The base image

FROM ubuntu:latest

# More instructions here that install software and copy files into the image.

COPY    /myapp/target/myapp.jar    /myapp/sriniapp.jar

# The command executed when running a Docker container based on this image.

CMD echo Starting Docker Container

Refer: https://www.youtube.com/watch?v=QLENvSMSD-0

5G

 5G is the 5th generation mobile network.

It is a new global wireless standard after 1G, 2G, 3G, and 4G networks. 5G enables a new kind of network that is designed to connect virtually everyone and everything together including machines, objects, and devices

Where 3G and 4G technologies have enabled us to browse the internet, use data driven services, increased bandwidths for streaming on Spotify or YouTube and so much more, 5G services are expected to revolutionize our lives. by enabling services that rely on advanced technologies like AR and VR, alongside cloud based gaming services like Google Stadia, NVidia GeForce Now and much more.

It is expected to be used in HD cameras that help improve safety and traffic management, smart grid control and smart retail too.

Previous generations:

First generation - 1G

1980s: 1G delivered analog voice.

Second generation - 2G

Early 1990s: 2G introduced digital voice (e.g. CDMA- Code Division Multiple Access).

Third generation - 3G

Early 2000s: 3G brought mobile data (e.g. CDMA2000).

Fourth generation - 4G LTE

2010s: 4G LTE ushered in the era of mobile broadband.

5G will help edge computing grow:

The 5G network’s speed should be 10 times faster than what the 4G network allows. That improvement opens possibilities for far-away sensors to instantly give updates about the connected devices. 

That increase in real-time processing will require new equipment with enhanced capabilities.

It’ll also raise the demand for edge computing since that method processes data at the borders of a network rather than transferring the data a significant distance first

To know more about Edge computing refer below link:

https://nextsrini.blogspot.com/2020/05/edge-computing.html

JIRA Rest API using Python

The Jira REST API enables you to interact with Jira programmatically. Use this API 

to build apps, script interactions with Jira, or develop any other type of 

integration.

The URIs for resources have the following structure:

https:///rest/api/3/

There are so many libraries like JIRA,request to access the JIRA services.
  

In the below code snippet, i had used request library and to save the results used 

Pandas library.

Pandas in the best library to save the results to Excel workbook.

Pandas library can be installed using either PIP/PIP3/easy-install. Make sure you 

are having right SSL certificate's before install pandas in intranet.

pip install pandas  
  
Sample Code Snippet:
  
import requests
from requests.auth import HTTPBasicAuth
import json
import pandas as pd

pd.set_option('display.max_columns', 500)
# url = "https://ABCD.atlassian.net/rest/api/3/issuetype"
url = "https://ABCD.atlassian.net/rest/api/3/search"

auth = HTTPBasicAuth("xxxxxxx", "yyyyyyyyyyyyyy")

headers = {
    "Accept": "application/json"
}
query = {
    'jql': 'project = ABCD'
}

response = requests.request(
    "GET",
    url,
    headers=headers,
    params=query,
    auth=auth
)
resdata = json.dumps(json.loads(response.text))
# print(json.dumps(json.loads(response.text), sort_keys=True, indent=4, 

separators=(",", ": ")))

if response.status_code != 200:
    print('Status:', response.status_code, 'Headers:', response.headers,
          'Error Response:', response.json())
    exit()
else:
    print('Server Connected.Please wait.....')

#for fetching all sprints data
df = json.loads(resdata)
maxresults = df['maxResult']
print(df['total'])
writer = pd.ExcelWriter('Jira_issues.xlsx', engine='xlsxwriter')
dk.to_excel(writer, sheet_name='Jira_Issues', index=False)
writer.save()