@sbalbach https://travis-ci.org/virtuald/pyhcl/jobs/555816019#L209 Looks like black doesn't like your code formatting. Mind running black on your PR and making the appropriate fixes?

(I have a lot of interest in this PR getting merged, hence the unsolicited drive-by review).

Sorry for the delayed response, I've been traveling.

Can you add some new simple tests for the new syntax that you're supporting? Otherwise someone is going to break it in the future.

I seem to have found an issue with EPLUS. I have reproduced it in the test_lexer.py by adding the following line to the COMPLEX_TOKEN_FIXTURES.

(["IDENTIFIER", "EQUAL", "LEFTBRACKET", "IDENTIFIER", "LEFTBRACKET", "NUMBER", "RIGHTBRACKET", "PERIOD", "EPLUS", "IDENTIFIER", "RIGHTBRACKET", ], "records = [aws_elasticsearch_domain.elasticsearch[0].endpoint]"),

That will pass the test even though there is no actual scientific notation in the string. The problem is that the '+' is optional. Since there is a period followed by an e, an EPLUS gets thrown into the mix which is causing me grief.

EPLUS is defined as:

r'(?<=\d|\.)[eE]\+?'

If I remove the ?, the EPLUS doesn't get added to the above test case but that will fail other tests because valid scientific notation is 0.e

I believe I've found a solution for the EPLUS issue:

r'(?<=\d)[eE]\+?|(?<=\d\.)[eE]\+?'

This passes all of the tests and doesn't give me an erroneous EPLUS from

records = [aws_elasticsearch_domain.elasticsearch[0].endpoint]

I'll work on new test cases next.

Good evening. I have tried to use this solution and faced a problem to parse .tf file with Conditional Operator

resource "openstack_compute_instance_v2" "instance_name" {
  availability_zone = var.cloud_release == "li" ? var.ghost-web_az : element(var.availability_zones, count.index)
}

Error is:

self = <hcl.parser.HclParser object at 0x10584ad68>, p = LexToken(EQ,'==',6,266)
E       ValueError: Line 6, column 266: unexpected EQ

Is something wrong with my syntax?

@PetrusHahol : So many permutations...I just uploaded a fix. Give it a try.

@PetrusHahol : So many permutations...I just uploaded a fix. Give it a try.

I've tested, now it works, thank you!

Thank you once again,
but I faced one more problem with converting from HCL to python object.

variable "list_variable" {
  type    = list(string)
  default = ["value1", "value2", "value3"]
}

After converting type looks like:

     "type": [   "list", "string"   ]

And terraform plan fails with error:

A type specification is either a primitive type keyword (bool, number, string)
or a complex type constructor call, like list(string).

According to documentation of tf0.12 it should be "type": "list(string)"

Also
template = file("templates/cluster.tpl")
name = format("instance%02d", count.index + 1)
web = join("\n", data.template_file.cluster.*.rendered)

According to documentation of tf0.12 it should be "type": "list(string)"

Can you post a link to the documentation that says this?

According to documentation of tf0.12 it should be "type": "list(string)"

Can you post a link to the documentation that says this?

Sure, https://www.terraform.io/docs/configuration/types.html

Also I faced lots of additional issues with hcl parser. Should I create issues in your repository? Or I can describe all of them here.
I fixed few of them in my forked repo.

According to documentation of tf0.12 it should be "type": "list(string)"

Can you post a link to the documentation that says this?

Sure, https://www.terraform.io/docs/configuration/types.html

Well, the terraform doc doesn't actually say how to convert from terraform to a python object because they don't deal with python objects.

I guess I'm confused on how you are using pyhcl. You're converting your terraform files to python objects and then running terraform on them somehow? Why aren't you just running terraform on your terraform files? I'm apparently missing something...

According to documentation of tf0.12 it should be "type": "list(string)"

Can you post a link to the documentation that says this?

Sure, https://www.terraform.io/docs/configuration/types.html

Well, the terraform doc doesn't actually say how to convert from terraform to a python object because they don't deal with python objects.

I guess I'm confused on how you are using pyhcl. You're converting your terraform files to python objects and then running terraform on them somehow? Why aren't you just running terraform on your terraform files? I'm apparently missing something...

So, in my solution(of my team) all terraform files in 11 version can be specified as .json and .tf as well and all of them written by users.

After terraform init we have internal logic which changes a lot of data in .tf files.

Unfortunately we can't write back .tf files after changes in hcl format because there is no libraries which allow to do it in python. Because of that we need to convert everything to .json

Now we are going to upgrade our solution to Terraform version 12 and still need to convert all .tf files to json before apply

Ok, I understand now. I've put a new version out there. Give it a try.

Is there an issue with ternary operations with spaces? Other than that and my comment above, this looks pretty good and I'd be happy to merge after those are addressed.

Some problems surfaced and few tests of ours are failed.
name = format("name_%02d", count.index + 1)
has been converted to:
"name": "format(name_%02d,count.index,+,1)"

There are two additional commas and no space between IDENTIFIER's. One of the commits how I added spaces

Also for all values without double quotes should be added interpolation
Tests are here

So, I have fixed all of issues. We have almost 5000 lines of hcl which converted with logic in my fork successfully. Once I will finally finish with testing, I will create pull request to @sbalbach fork.

@virtuald It would be great to create release or tag before this pull will be merged because there is no backward compatibility.

I'm going to be on vacation for the next week+ so I've put out the changes that I had which should address what has already been pointed out.

@PetrusHahol this particular PR doesn't seem to break backwards compatibility as far as I can see (as all of the old test fixtures seem to work?).

I haven't seen any backward compatibility issues either. I'd suggest a major version change, though.

So, terraform 11 syntax contained interpolation everywhere in double quotes. Before(in tf11) every time we injected another variable via "${expression}"
With terraform 12 they deleted double quotes and interpolation as well, but it doesn't work with json. In json syntax we need to add interpolation back. Few examples :

• Example1:
  Before:
  hcl resource_name_output = ["${resource1.name1}", "${resource2.name2}"]
  json { "resource_name_output" : ["${resource1.name1}", "${resource2.name2}"] }
  Now:
  hcl resource_name_output = [resource1.name1, resource2.name2]
  json { "resource_name_output" : ["${resource1.name1}", "${resource2.name2}"] } <= interpolation has to appear.

• Example2:
  Before:
  hcl bar = "${file("bing/bong.txt")}"
  json { "bar": "${file("bing/bong.txt")}" }
  Now:
  hcl bar = file("bing/bong.txt")
  json { "bar": "${file(\"bing/bong.txt\")" } <= interpolation has to appear.

• Example3:
  Before:
  hcl cluster_3 = "${format("name_%02d", count.index + 1)}"
  json { "cluster_3 ": "${format(\"name_%02d\", count.index + 1)}"
  Now:
  hcl cluster_3 = format("name_%02d", count.index + 1)
  json { "cluster_3 ": "${format(\"name_%02d\", count.index + 1)}" <= interpolation has to appear.

• Example4:
  Before:
  hcl value = "string"
  json { "value ": "string"}
  Now:
  hcl value = "string"
  json { "value ": "string" } <= no interpolation (just a string)

Two exceptions :

1. type of variable list(string), map(string), etc should be without interpolation
2. depends_on values should be without interpolation

If interpolation will not be added, terraform will work with values as strings.
https://github.com/PetrusHahol/pyhcl I changed a lot of tests, added couple of new and added additional token - STRING_IDENTIFIER with r'"(?:[^\\"]|\.)*"' regular expression. We need to separate string with double quotes and without.

Pull request to @sbalbach fork is here .

@PetrusHahol I'm a bit confused by your examples, but I suppose that I don't really track HCL anymore. Keep in mind this is not a terraform library, it is an HCL library (for example, other hashicorp products such as nomad use HCL). Things like interpolation/etc are specific to terraform, and would belong in a pyterraform library.

@PetrusHahol I'm a bit confused by your examples, but I suppose that I don't really track HCL anymore. Keep in mind this is not a terraform library, it is an HCL library (for example, other hashicorp products such as nomad use HCL). Things like interpolation/etc are specific to terraform, and would belong in a pyterraform library.

Just realised that changes I did it's for hcl2

From my point of view, It means this pull requests should not be here. Terraform 12 uses fully another hcl parser (hcl2).

@PetrusHahol this is for the changes brought about by terraform 0.12. As I said in issue #57 Hashicorp plans at some future date to combine HCL & HCL2. The changes that I had made were all backward compatible with HCL. I don't use the conversion to json option, though; I use the python objects in memory that pyhcl creates. I haven't had a chance since I got back from vacation to try your changes with my other packages to see if or how badly they get broken. Hopefully, I will have a chance to do that Monday.

@PetrusHahol I've run into a problem with your STRING_IDENTIFER on the below line:

value = "urn:amazon:cognito:sp:${join("", aws_cognito_user_pool.elasticsearch.*.id)}"

It matches urn:amazon:cognito:sp:${join( as a STRING_IDENTIFIER and another one as , aws_cognito_user_pool.elasticsearch..id)} when there should be only one that is the entire urn:amazon:cognito:sp:${join("", aws_cognito_user_pool.elasticsearch..id)} because the inner "" is inside of the ${}. I'm not good enough at regex to figure out how to define that. :(

@PetrusHahol it seems that you are basically transforming from terraform 0.12 to the old terraform 0.11 syntax. I don't believe that is the correct way to handle it.
Also, if you are transforming terraform to json, be aware of this issue.

@sbalbach Sorry that didn't answer long time. So, basically you are right about converting to old syntax, but it seems like this is right way. I will check your example which does not work, but it seems you just need to add \ before " and it will look like

value = "urn:amazon:cognito:sp:${join(\"\", aws_cognito_user_pool.elasticsearch.*.id)}"

@sbalbach Sorry that didn't answer long time. So, basically you are right about converting to old syntax, but it seems like this is right way. I will check your example which does not work, but it seems you just need to add \ before " and it will look like

value = "urn:amazon:cognito:sp:${join(\"\", aws_cognito_user_pool.elasticsearch.*.id)}"

The hcl is valid as it is. I can't force the users to conform to this tool if they are writing valid hcl.

Hello, what is the status of this PR? Thanks :-)

I have given up trying to make this work for everyone else's needs. I have a version that works for what I need but breaks with no fix in sight if I make changes to address some of the later issues that were brought up.

I've made another package just for hcl2 that could help https://pypi.org/project/python-hcl2/. Probably doesn't cover every use case yet though

Fixed in #63