[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing automation #235

Open
jzaefferer opened this issue Apr 16, 2014 · 24 comments
Open

Testing automation #235

jzaefferer opened this issue Apr 16, 2014 · 24 comments

Comments

@jzaefferer
Copy link
Contributor

We want to test this in all supported browsers automatically. I'll look into creating a configuration file for browserstack-runner to see if that works here. Otherwise we can try it with TestSwarm.

@jzaefferer
Copy link
Contributor Author

@rxaviers can you follow the usage instructions on here? https://github.com/browserstack/browserstack-runner

It should be really easy to get running. Just make sure you've got BROWSERSTACK_USER and BROWSERSTACK_KEY set as env vars.

Would be good to get some feedback from someone who hasn't used the tool before. Please let me know about any issues, no matter how tiny.

@rxaviers
Copy link
Member

I assume you have used it before then. Do you have any project with this setup?

I've come up with the below browserstack.json:

{
  "username": "<username>",
  "key": "<key>",
  "test_framework": "qunit",
  "test_path": ["test"],
  "browsers": [
    "chrome_previous",
    "chrome_latest",
    "firefox_previous",
    "firefox_latest",
    "ie_8",
    "ie_9",
    "ie_10",
    "ie_11",
    "ie_latest",
    "opera_12_1",
    "opera_previous",
    "opera_latest",
    "safari_5_1",
    "safari_previous",
    "safari_latest"
  ]
}

Questions:
1)

  "test_framework": "qunit",
  "test_path": ["test"],

Tests should be executed using grunt test. Is the above equivalent?

  "browsers": [...]

Globalize supports http://jquery.com/browser-support/, except IE6 and IE7 right? What are the keywords for mobile? Is config missing any browser entry?

@jzaefferer
Copy link
Contributor Author
  1. Let's get this running first using the browserstack-runner CLI, we can worry about grunt integration later.

  2. You don't need both ie_11 and ie_latest. As for mobile browsers, we can add those later, once this is working. Meanwhile I've asked them about the lacks of docs for mobile browser config.

@rxaviers
Copy link
Member

Note I am not worried about grunt integration. My concern is whether or not executing browserstack-runner CLI will run the same tests we do via grunt test.

@rxaviers
Copy link
Member

You don't need both ie_11 and ie_latest.

Why?

As for mobile browsers, we can add those later, once this is working. Meanwhile I've asked them about the lacks of docs for mobile browser config.

Excellent!

@jzaefferer
Copy link
Contributor Author

I lost track of our IRC conversation. What was your last result after changing the test_path?

@rxaviers
Copy link
Member

@jzaefferer
Copy link
Contributor Author

I just pulled and ran grunt and got a timeout from PhantomJS:

Running "qunit:unit" (qunit) task
Testing http://localhost:9001/unit.html
>> PhantomJS timed out, possibly due to a missing QUnit start() call.
Warning: 1/1 assertions failed (0ms) Use --force to continue.

Seems pointless to run tests through browserstack when they don't even run through PhantomJS.

Once that works, could you retry with browserstack-runner? I suggest to start with just one browser, that will speed things up a lot.

@rxaviers
Copy link
Member

It works just fine to me... I have double checked by cloning Globalize into a brand new directory, npm and bower installed dependencies, ran grunt and it worked again. This is my environment https://gist.github.com/rxaviers/11368480.

Could you double check yours please?

PS: What command does browserstack-runner actually run?

@jzaefferer
Copy link
Contributor Author

I've tracked down the problem within browserstack-runner and sent a PR: browserstack/browserstack-runner#83

Once that is merged and released it should work fine. I had a green run locally without timeouts.

@rxaviers
Copy link
Member

👍

@rxaviers
Copy link
Member

@jzaefferer your PR is merged. Are we good to proceed?

@jzaefferer
Copy link
Contributor Author

Sure. It was working for me with the patch, so can you give it another try?

@rxaviers
Copy link
Member

I've updated my browserstack-runner to 0.1.14. But, I'm getting this error

$ browserstack-runner

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: EACCES, open '/usr/lib/node_modules/browserstack-runner/lib/BrowserStackLocal'

There's no BrowserStackLocal file in my filesystem. Although, I do find the /usr/lib/node_modules/browserstack-runner/lib path.

@rxaviers
Copy link
Member

I have pushed my code in here (which is just the config file) https://github.com/jquery/globalize/compare/fix-235-testing-automation?expand=1 in case you need to see it.

@arschmitz
Copy link

@rxaviers I ran into this same error working on this for mobile after we talked tonight. Got up to 6 files running and passing in mobile but when I added the file https://github.com/jquery/jquery-mobile/blob/master/tests/unit/page-sections/index.html i started getting this error

@rxaviers
Copy link
Member

Thanks @arschmitz for letting me know

@jzaefferer
Copy link
Contributor Author

Oh well. Can either of you report that against the project itself?

@rxaviers
Copy link
Member
rxaviers commented Jul 9, 2014

Include mobile browsers #278

@rxaviers rxaviers added this to the 1.0.0 milestone Aug 27, 2014
@rxaviers
Copy link
Member

For the record, that's the issue we had browserstack/browserstack-runner#91 .

Now, browserstack-runner works and it submits the tests just fine. Although, the results are inconsistent, ie. tests fail intermittently when ran automatically. But, they pass when ran manually.

@jzaefferer
Copy link
Contributor Author

#370 fixed several issues highlighted by running through browserstack-runner. Adding waitTimeout: 30 to the requirejs config proved to fix the timeout issues, while also speeding up runtime of passing tests (still don't know why tests run faster with a longer timeout).

#367 (browserstack.json) still needs some work, should at least add functional tests to the mix.

jzaefferer added a commit that referenced this issue Jan 7, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes #235
jzaefferer added a commit that referenced this issue Jan 12, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes #235
jzaefferer added a commit that referenced this issue Jan 12, 2015
Uses a separate config file for IE8:

BROWSERSTACK_JSON=browserstack-ie8.json browserstack-runner

Ref #235
jzaefferer added a commit to jzaefferer/globalize that referenced this issue Jan 12, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes globalizejs#235
jzaefferer added a commit to jzaefferer/globalize that referenced this issue Jan 12, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes globalizejs#235
jzaefferer added a commit to jzaefferer/globalize that referenced this issue Jan 12, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes globalizejs#235
jzaefferer added a commit to jzaefferer/globalize that referenced this issue Jan 13, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes globalizejs#235
jzaefferer added a commit to jzaefferer/globalize that referenced this issue Jan 13, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes globalizejs#235
jzaefferer added a commit to jzaefferer/globalize that referenced this issue Jan 14, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes globalizejs#235
jzaefferer added a commit to jzaefferer/globalize that referenced this issue Jan 21, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes globalizejs#235
@rxaviers rxaviers modified the milestones: 1.1.0, 1.0.0 Feb 12, 2015
jzaefferer added a commit to jzaefferer/globalize that referenced this issue Feb 17, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes globalizejs#235
jzaefferer added a commit to jzaefferer/globalize that referenced this issue Feb 24, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes globalizejs#235
jzaefferer added a commit to jzaefferer/globalize that referenced this issue Apr 10, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes globalizejs#235
jzaefferer added a commit to jzaefferer/globalize that referenced this issue Apr 10, 2015
As for the obscured credentials: Those are easy enough to decode, the
point is to avoid them being searchable, that is a search for the
relevant keywords on GitHub won't immediately show these credentials.
This is a temporary solution until Travis CI implements a proper one.

Fixes globalizejs#235
@rxaviers
Copy link
Member

WIP PR #450

@rxaviers
Copy link
Member

Close in favor of #450

@rxaviers
Copy link
Member

Actually, reopen. Should be fixed by #450.

@rxaviers rxaviers reopened this Oct 10, 2015
@rxaviers rxaviers modified the milestone: 1.1.0 Oct 11, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants